Nov 03 08:36:48.818463 ip-10-0-139-73 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 03 08:36:48.818472 ip-10-0-139-73 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 03 08:36:48.818478 ip-10-0-139-73 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 03 08:36:48.818694 ip-10-0-139-73 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 03 08:36:58.999160 ip-10-0-139-73 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 03 08:36:58.999177 ip-10-0-139-73 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 0992e8ce42694c76bfa99413680324e8 --
Nov 03 08:38:55.125215 ip-10-0-139-73 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 08:38:55.525189 ip-10-0-139-73 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 08:38:55.525189 ip-10-0-139-73 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 08:38:55.525189 ip-10-0-139-73 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 08:38:55.525189 ip-10-0-139-73 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 08:38:55.528595 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528514    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 08:38:55.528595 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528594    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 08:38:55.528595 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528597    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 08:38:55.528685 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528601    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 08:38:55.528685 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528603    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 08:38:55.528685 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528606    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 08:38:55.528685 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528609    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 08:38:55.528685 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528612    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 08:38:55.528685 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528614    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 08:38:55.528685 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528617    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 08:38:55.528685 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528620    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 08:38:55.528685 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528622    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 08:38:55.528685 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528625    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 08:38:55.528685 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528629    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 08:38:55.528685 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528647    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 08:38:55.528685 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528651    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 08:38:55.528685 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528654    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 08:38:55.528685 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528657    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 08:38:55.528685 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528659    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 08:38:55.528685 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528662    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 08:38:55.528685 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528664    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 08:38:55.528685 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528667    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 08:38:55.529143 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528670    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 08:38:55.529143 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528673    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 08:38:55.529143 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528675    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 08:38:55.529143 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528678    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 08:38:55.529143 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528680    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 08:38:55.529143 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528683    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 08:38:55.529143 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528686    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 08:38:55.529143 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528688    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 08:38:55.529143 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528691    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 08:38:55.529143 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528694    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 08:38:55.529143 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528697    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 08:38:55.529143 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528699    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 08:38:55.529143 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528702    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 08:38:55.529143 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528704    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 08:38:55.529143 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528707    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 08:38:55.529143 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528709    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 08:38:55.529143 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528711    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 08:38:55.529143 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528715    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 08:38:55.529143 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528717    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 08:38:55.529590 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528720    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 08:38:55.529590 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528722    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 08:38:55.529590 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528725    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 08:38:55.529590 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528727    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 08:38:55.529590 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528730    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 08:38:55.529590 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528732    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 08:38:55.529590 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528735    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 08:38:55.529590 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528737    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 08:38:55.529590 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528740    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 08:38:55.529590 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528742    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 08:38:55.529590 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528746    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 08:38:55.529590 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528749    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 08:38:55.529590 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528752    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 08:38:55.529590 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528755    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 08:38:55.529590 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528759    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 08:38:55.529590 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528761    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 08:38:55.529590 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528764    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 08:38:55.529590 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528767    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 08:38:55.529590 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528770    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 08:38:55.530047 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528772    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 08:38:55.530047 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528774    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 08:38:55.530047 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528777    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 08:38:55.530047 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528779    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 08:38:55.530047 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528782    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 08:38:55.530047 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528784    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 08:38:55.530047 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528786    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 08:38:55.530047 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528789    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 08:38:55.530047 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528791    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 08:38:55.530047 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528794    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 08:38:55.530047 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528796    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 08:38:55.530047 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528798    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 08:38:55.530047 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528802    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 08:38:55.530047 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528804    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 08:38:55.530047 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528807    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 08:38:55.530047 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528811    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 08:38:55.530047 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528813    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 08:38:55.530047 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528816    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 08:38:55.530047 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528818    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 08:38:55.530494 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528821    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 08:38:55.530494 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528824    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 08:38:55.530494 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528826    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 08:38:55.530494 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528829    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 08:38:55.530494 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528831    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 08:38:55.530494 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528834    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 08:38:55.530494 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528836    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 08:38:55.530494 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528839    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 08:38:55.530494 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528842    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 08:38:55.530494 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528845    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 08:38:55.530494 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528848    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 08:38:55.530494 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528850    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 08:38:55.530494 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528853    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 08:38:55.530494 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528855    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 08:38:55.530494 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528858    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 08:38:55.530494 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528860    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 08:38:55.530494 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528863    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 08:38:55.530494 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528865    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 08:38:55.530494 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528868    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 08:38:55.530494 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528870    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 08:38:55.531117 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528873    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 08:38:55.531117 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528876    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 08:38:55.531117 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528879    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 08:38:55.531117 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528881    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 08:38:55.531117 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528884    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 08:38:55.531117 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528886    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 08:38:55.531117 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.528889    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 08:38:55.531117 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529672    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 08:38:55.531117 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529679    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 08:38:55.531117 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529684    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 08:38:55.531117 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529688    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 08:38:55.531117 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529691    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 08:38:55.531117 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529694    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 08:38:55.531117 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529696    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 08:38:55.531117 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529699    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 08:38:55.531117 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529701    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 08:38:55.531117 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529704    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 08:38:55.531117 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529706    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 08:38:55.531117 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529709    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 08:38:55.531117 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529711    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 08:38:55.531604 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529714    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 08:38:55.531604 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529716    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 08:38:55.531604 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529719    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 08:38:55.531604 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529722    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 08:38:55.531604 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529725    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 08:38:55.531604 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529727    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 08:38:55.531604 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529730    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 08:38:55.531604 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529733    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 08:38:55.531604 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529735    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 08:38:55.531604 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529738    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 08:38:55.531604 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529741    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 08:38:55.531604 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529743    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 08:38:55.531604 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529746    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 08:38:55.531604 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529748    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 08:38:55.531604 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529751    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 08:38:55.531604 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529753    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 08:38:55.531604 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529755    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 08:38:55.531604 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529758    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 08:38:55.531604 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529760    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 08:38:55.531604 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529765    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 08:38:55.532092 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529767    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 08:38:55.532092 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529770    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 08:38:55.532092 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529773    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 08:38:55.532092 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529775    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 08:38:55.532092 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529778    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 08:38:55.532092 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529780    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 08:38:55.532092 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529783    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 08:38:55.532092 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529785    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 08:38:55.532092 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529787    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 08:38:55.532092 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529791    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 08:38:55.532092 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529793    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 08:38:55.532092 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529796    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 08:38:55.532092 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529799    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 08:38:55.532092 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529801    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 08:38:55.532092 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529804    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 08:38:55.532092 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529806    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 08:38:55.532092 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529809    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 08:38:55.532092 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529811    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 08:38:55.532092 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529814    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 08:38:55.532550 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529816    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 08:38:55.532550 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529819    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 08:38:55.532550 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529821    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 08:38:55.532550 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529824    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 08:38:55.532550 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529826    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 08:38:55.532550 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529828    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 08:38:55.532550 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529831    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 08:38:55.532550 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529833    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 08:38:55.532550 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529836    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 08:38:55.532550 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529838    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 08:38:55.532550 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529841    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 08:38:55.532550 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529843    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 08:38:55.532550 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529846    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 08:38:55.532550 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529849    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 08:38:55.532550 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529852    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 08:38:55.532550 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529854    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 08:38:55.532550 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529857    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 08:38:55.532550 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529860    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 08:38:55.532550 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529862    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 08:38:55.532550 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529865    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 08:38:55.533046 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529867    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 08:38:55.533046 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529870    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 08:38:55.533046 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529872    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 08:38:55.533046 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529874    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 08:38:55.533046 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529877    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 08:38:55.533046 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529879    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 08:38:55.533046 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529882    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 08:38:55.533046 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529884    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 08:38:55.533046 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529886    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 08:38:55.533046 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529889    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 08:38:55.533046 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529891    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 08:38:55.533046 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529894    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 08:38:55.533046 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529896    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 08:38:55.533046 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529899    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 08:38:55.533046 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529902    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 08:38:55.533046 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529905    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 08:38:55.533046 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529907    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 08:38:55.533046 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529910    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 08:38:55.533046 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529912    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 08:38:55.533508 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529914    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 08:38:55.533508 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529917    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 08:38:55.533508 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529919    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 08:38:55.533508 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529921    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 08:38:55.533508 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529924    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 08:38:55.533508 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529926    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 08:38:55.533508 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529929    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 08:38:55.533508 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529932    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 08:38:55.533508 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529935    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 08:38:55.533508 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529939    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 08:38:55.533508 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529959    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 08:38:55.533508 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529962    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 08:38:55.533508 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529966    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 08:38:55.533508 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529969    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 08:38:55.533508 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.529971    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 08:38:55.533508 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530459    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 08:38:55.533508 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530464    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 08:38:55.533508 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530468    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 08:38:55.533508 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530471    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 08:38:55.534459 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530474    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 08:38:55.534459 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530477    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 08:38:55.534459 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530480    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 08:38:55.534459 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530482    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 08:38:55.534459 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530485    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 08:38:55.534459 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530488    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 08:38:55.534459 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530490    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 08:38:55.534459 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530493    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 08:38:55.534459 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530496    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 08:38:55.534459 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530499    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 08:38:55.534459 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530501    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 08:38:55.534459 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530504    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 08:38:55.534459 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530507    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 08:38:55.534459 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530510    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 08:38:55.534459 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530512    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 08:38:55.534459 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530515    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 08:38:55.534459 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530517    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 08:38:55.534459 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530520    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 08:38:55.534459 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530523    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 08:38:55.534459 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530525    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 08:38:55.534937 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530537    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 08:38:55.534937 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530541    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 08:38:55.534937 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530544    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 08:38:55.534937 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530546    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 08:38:55.534937 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530549    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 08:38:55.534937 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530551    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 08:38:55.534937 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530554    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 08:38:55.534937 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530557    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 08:38:55.534937 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530559    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 08:38:55.534937 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530562    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 08:38:55.534937 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530564    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 08:38:55.534937 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530567    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 08:38:55.534937 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530569    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 08:38:55.534937 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530572    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 08:38:55.534937 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530574    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 08:38:55.534937 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530576    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 08:38:55.534937 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530579    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 08:38:55.534937 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530581    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 08:38:55.534937 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530584    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 08:38:55.534937 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530586    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 08:38:55.535459 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530589    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 08:38:55.535459 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530591    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 08:38:55.535459 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530594    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 08:38:55.535459 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530596    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 08:38:55.535459 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530599    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 08:38:55.535459 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530602    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 08:38:55.535459 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530605    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 08:38:55.535459 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530607    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 08:38:55.535459 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530611    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 08:38:55.535459 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530615    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 08:38:55.535459 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530617    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 08:38:55.535459 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530620    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 08:38:55.535459 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530623    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 08:38:55.535459 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530629    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 08:38:55.535459 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530632    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 08:38:55.535459 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530634    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 08:38:55.535459 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530637    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 08:38:55.535459 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530640    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 08:38:55.535459 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530642    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 08:38:55.535908 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530645    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 08:38:55.535908 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530647    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 08:38:55.535908 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530650    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 08:38:55.535908 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530652    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 08:38:55.535908 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530655    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 08:38:55.535908 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530673    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 08:38:55.535908 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530677    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 08:38:55.535908 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530680    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 08:38:55.535908 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530683    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 08:38:55.535908 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530685    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 08:38:55.535908 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530688    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 08:38:55.535908 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530690    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 08:38:55.535908 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530693    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 08:38:55.535908 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530695    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 08:38:55.535908 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530698    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 08:38:55.535908 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530701    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 08:38:55.535908 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530704    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 08:38:55.535908 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530707    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 08:38:55.535908 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530710    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 08:38:55.536436 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530712    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 08:38:55.536436 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530714    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 08:38:55.536436 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530717    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 08:38:55.536436 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530720    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 08:38:55.536436 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530723    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 08:38:55.536436 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530725    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 08:38:55.536436 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530727    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 08:38:55.536436 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530731    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 08:38:55.536436 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530734    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 08:38:55.536436 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530737    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 08:38:55.536436 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530739    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 08:38:55.536436 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530742    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 08:38:55.536436 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530744    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 08:38:55.536436 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530747    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 08:38:55.536436 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530749    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 08:38:55.536436 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530753    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 08:38:55.536436 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530756    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 08:38:55.536436 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530761    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 08:38:55.536436 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530764    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 08:38:55.536895 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530766    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 08:38:55.536895 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530769    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 08:38:55.536895 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530772    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 08:38:55.536895 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530774    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 08:38:55.536895 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.530777    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 08:38:55.536895 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531737    2575 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 08:38:55.536895 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531748    2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 08:38:55.536895 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531755    2575 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 08:38:55.536895 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531759    2575 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 08:38:55.536895 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531764    2575 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 08:38:55.536895 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531767    2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 08:38:55.536895 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531771    2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 08:38:55.536895 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531776    2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 08:38:55.536895 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531780    2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 08:38:55.536895 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531783    2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 08:38:55.536895 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531786    2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 08:38:55.536895 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531790    2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 08:38:55.536895 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531793    2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 08:38:55.536895 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531796    2575 flags.go:64] FLAG: --cgroup-root=""
Nov 03 08:38:55.536895 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531799    2575 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 08:38:55.536895 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531802    2575 flags.go:64] FLAG: --client-ca-file=""
Nov 03 08:38:55.536895 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531805    2575 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 08:38:55.536895 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531808    2575 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 08:38:55.537448 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531812    2575 flags.go:64] FLAG: --cluster-domain=""
Nov 03 08:38:55.537448 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531816    2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 08:38:55.537448 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531819    2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 08:38:55.537448 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531822    2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 08:38:55.537448 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531825    2575 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 08:38:55.537448 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531830    2575 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 08:38:55.537448 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531832    2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 08:38:55.537448 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531835    2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 08:38:55.537448 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531839    2575 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 08:38:55.537448 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531841    2575 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 08:38:55.537448 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531844    2575 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 08:38:55.537448 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531847    2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 08:38:55.537448 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531851    2575 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 08:38:55.537448 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531854    2575 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 08:38:55.537448 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531858    2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 08:38:55.537448 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531861    2575 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 08:38:55.537448 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531864    2575 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 08:38:55.537448 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531866    2575 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 08:38:55.537448 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531869    2575 flags.go:64] FLAG: --enable-server="true"
Nov 03 08:38:55.537448 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531872    2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 08:38:55.537448 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531877    2575 flags.go:64] FLAG: --event-burst="100"
Nov 03 08:38:55.537448 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531880    2575 flags.go:64] FLAG: --event-qps="50"
Nov 03 08:38:55.537448 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531883    2575 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 08:38:55.537448 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531885    2575 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 08:38:55.538041 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531888    2575 flags.go:64] FLAG: --eviction-hard=""
Nov 03 08:38:55.538041 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531893    2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 08:38:55.538041 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531895    2575 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 08:38:55.538041 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531898    2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 08:38:55.538041 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531901    2575 flags.go:64] FLAG: --eviction-soft=""
Nov 03 08:38:55.538041 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531904    2575 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 08:38:55.538041 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531907    2575 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 08:38:55.538041 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531910    2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 08:38:55.538041 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531913    2575 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 08:38:55.538041 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531916    2575 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 08:38:55.538041 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531919    2575 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 08:38:55.538041 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531922    2575 flags.go:64] FLAG: --feature-gates=""
Nov 03 08:38:55.538041 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531926    2575 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 08:38:55.538041 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531929    2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 08:38:55.538041 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531932    2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 08:38:55.538041 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531936    2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 08:38:55.538041 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531939    2575 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 08:38:55.538041 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531952    2575 flags.go:64] FLAG: --help="false"
Nov 03 08:38:55.538041 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531955    2575 flags.go:64] FLAG: --hostname-override="ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:38:55.538041 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531959    2575 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 08:38:55.538041 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531962    2575 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 08:38:55.538041 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531964    2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 08:38:55.538041 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531968    2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 08:38:55.538613 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531971    2575 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 08:38:55.538613 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531974    2575 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 08:38:55.538613 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531977    2575 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 08:38:55.538613 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531980    2575 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 08:38:55.538613 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531983    2575 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 08:38:55.538613 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531986    2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 08:38:55.538613 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531988    2575 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 08:38:55.538613 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531991    2575 flags.go:64] FLAG: --kube-reserved=""
Nov 03 08:38:55.538613 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531994    2575 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 08:38:55.538613 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.531997    2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 08:38:55.538613 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532000    2575 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 08:38:55.538613 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532003    2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 08:38:55.538613 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532005    2575 flags.go:64] FLAG: --lock-file=""
Nov 03 08:38:55.538613 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532008    2575 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 08:38:55.538613 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532011    2575 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 08:38:55.538613 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532014    2575 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 08:38:55.538613 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532020    2575 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 08:38:55.538613 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532023    2575 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 08:38:55.538613 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532026    2575 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 08:38:55.538613 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532029    2575 flags.go:64] FLAG: --logging-format="text"
Nov 03 08:38:55.538613 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532032    2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 08:38:55.538613 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532036    2575 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 08:38:55.538613 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532038    2575 flags.go:64] FLAG: --manifest-url=""
Nov 03 08:38:55.538613 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532041    2575 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 08:38:55.538613 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532046    2575 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 08:38:55.539264 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532049    2575 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 08:38:55.539264 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532052    2575 flags.go:64] FLAG: --max-pods="110"
Nov 03 08:38:55.539264 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532055    2575 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 08:38:55.539264 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532058    2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 08:38:55.539264 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532061    2575 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 08:38:55.539264 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532064    2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 08:38:55.539264 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532067    2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 08:38:55.539264 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532070    2575 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 08:38:55.539264 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532073    2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 08:38:55.539264 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532080    2575 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 08:38:55.539264 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532083    2575 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 08:38:55.539264 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532086    2575 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 08:38:55.539264 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532089    2575 flags.go:64] FLAG: --pod-cidr=""
Nov 03 08:38:55.539264 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532092    2575 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 08:38:55.539264 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532095    2575 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 08:38:55.539264 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532098    2575 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 08:38:55.539264 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532101    2575 flags.go:64] FLAG: --port="10250"
Nov 03 08:38:55.539264 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532105    2575 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 08:38:55.539264 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532108    2575 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-025d50e6e4ea3a220"
Nov 03 08:38:55.539264 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532111    2575 flags.go:64] FLAG: --qos-reserved=""
Nov 03 08:38:55.539264 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532113    2575 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 08:38:55.539264 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532116    2575 flags.go:64] FLAG: --register-node="true"
Nov 03 08:38:55.539264 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532119    2575 flags.go:64] FLAG: --register-with-taints=""
Nov 03 08:38:55.539264 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532123    2575 flags.go:64] FLAG: --registry-burst="10"
Nov 03 08:38:55.539264 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532125    2575 flags.go:64] FLAG: --registry-qps="5"
Nov 03 08:38:55.539264 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532128    2575 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 08:38:55.539902 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532132    2575 flags.go:64] FLAG: --reserved-memory=""
Nov 03 08:38:55.539902 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532136    2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 08:38:55.539902 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532139    2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 08:38:55.539902 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532142    2575 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 08:38:55.539902 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532144    2575 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 08:38:55.539902 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532147    2575 flags.go:64] FLAG: --runonce="false"
Nov 03 08:38:55.539902 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532150    2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 08:38:55.539902 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532153    2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 08:38:55.539902 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532156    2575 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 08:38:55.539902 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532159    2575 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 08:38:55.539902 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532162    2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 08:38:55.539902 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532165    2575 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 08:38:55.539902 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532170    2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 08:38:55.539902 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532173    2575 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 08:38:55.539902 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532176    2575 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 08:38:55.539902 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532179    2575 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 08:38:55.539902 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532182    2575 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 08:38:55.539902 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532184    2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 08:38:55.539902 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532188    2575 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 08:38:55.539902 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532191    2575 flags.go:64] FLAG: --system-cgroups=""
Nov 03 08:38:55.539902 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532194    2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 08:38:55.539902 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532199    2575 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 08:38:55.539902 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532203    2575 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 08:38:55.539902 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532206    2575 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 08:38:55.539902 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532210    2575 flags.go:64] FLAG: --tls-min-version=""
Nov 03 08:38:55.540524 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532213    2575 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 08:38:55.540524 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532216    2575 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 08:38:55.540524 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532219    2575 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 08:38:55.540524 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532222    2575 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 08:38:55.540524 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532225    2575 flags.go:64] FLAG: --v="2"
Nov 03 08:38:55.540524 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532229    2575 flags.go:64] FLAG: --version="false"
Nov 03 08:38:55.540524 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532233    2575 flags.go:64] FLAG: --vmodule=""
Nov 03 08:38:55.540524 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532240    2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 08:38:55.540524 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532244    2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 08:38:55.540524 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532410    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 08:38:55.540524 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532415    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 08:38:55.540524 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532418    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 08:38:55.540524 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532421    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 08:38:55.540524 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532424    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 08:38:55.540524 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532426    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 08:38:55.540524 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532429    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 08:38:55.540524 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532432    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 08:38:55.540524 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532435    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 08:38:55.540524 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532437    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 08:38:55.540524 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532442    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 08:38:55.540524 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532444    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 08:38:55.540524 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532447    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 08:38:55.541061 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532449    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 08:38:55.541061 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532452    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 08:38:55.541061 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532454    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 08:38:55.541061 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532457    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 08:38:55.541061 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532460    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 08:38:55.541061 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532462    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 08:38:55.541061 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532465    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 08:38:55.541061 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532469    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 08:38:55.541061 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532471    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 08:38:55.541061 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532474    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 08:38:55.541061 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532476    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 08:38:55.541061 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532479    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 08:38:55.541061 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532481    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 08:38:55.541061 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532484    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 08:38:55.541061 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532487    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 08:38:55.541061 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532491    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 08:38:55.541061 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532495    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 08:38:55.541061 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532498    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 08:38:55.541061 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532501    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 08:38:55.541524 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532504    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 08:38:55.541524 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532507    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 08:38:55.541524 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532509    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 08:38:55.541524 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532512    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 08:38:55.541524 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532514    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 08:38:55.541524 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532517    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 08:38:55.541524 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532520    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 08:38:55.541524 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532522    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 08:38:55.541524 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532524    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 08:38:55.541524 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532527    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 08:38:55.541524 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532530    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 08:38:55.541524 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532533    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 08:38:55.541524 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532535    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 08:38:55.541524 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532538    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 08:38:55.541524 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532540    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 08:38:55.541524 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532543    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 08:38:55.541524 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532545    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 08:38:55.541524 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532547    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 08:38:55.541524 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532550    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 08:38:55.541524 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532552    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 08:38:55.541993 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532557    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 08:38:55.541993 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532560    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 08:38:55.541993 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532562    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 08:38:55.541993 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532565    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 08:38:55.541993 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532568    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 08:38:55.541993 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532570    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 08:38:55.541993 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532573    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 08:38:55.541993 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532575    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 08:38:55.541993 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532578    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 08:38:55.541993 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532580    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 08:38:55.541993 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532583    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 08:38:55.541993 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532585    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 08:38:55.541993 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532588    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 08:38:55.541993 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532591    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 08:38:55.541993 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532594    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 08:38:55.541993 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532598    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 08:38:55.541993 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532601    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 08:38:55.541993 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532603    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 08:38:55.541993 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532606    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 08:38:55.541993 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532608    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 08:38:55.542486 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532611    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 08:38:55.542486 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532613    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 08:38:55.542486 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532616    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 08:38:55.542486 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532619    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 08:38:55.542486 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532621    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 08:38:55.542486 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532624    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 08:38:55.542486 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532627    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 08:38:55.542486 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532629    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 08:38:55.542486 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532631    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 08:38:55.542486 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532634    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 08:38:55.542486 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532636    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 08:38:55.542486 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532639    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 08:38:55.542486 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532643    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 08:38:55.542486 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532645    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 08:38:55.542486 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532648    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 08:38:55.542486 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532650    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 08:38:55.542486 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532652    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 08:38:55.542486 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532655    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 08:38:55.542486 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532657    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 08:38:55.543255 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532660    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 08:38:55.543255 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532662    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 08:38:55.543255 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532665    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 08:38:55.543255 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532668    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 08:38:55.543255 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532670    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 08:38:55.543255 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532672    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 08:38:55.543255 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532675    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 08:38:55.543255 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532678    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 08:38:55.543255 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532680    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 08:38:55.543255 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532683    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 08:38:55.543255 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532685    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 08:38:55.543255 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532687    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 08:38:55.543255 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532690    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 08:38:55.543255 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532692    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 08:38:55.543255 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532694    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 08:38:55.543255 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.532723    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 08:38:55.544221 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532772    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 08:38:55.544221 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532777    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 08:38:55.544221 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532780    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 08:38:55.544221 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532783    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 08:38:55.544221 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532786    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 08:38:55.544221 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532788    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 08:38:55.544221 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532791    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 08:38:55.544221 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532794    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 08:38:55.544221 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532796    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 08:38:55.544221 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532799    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 08:38:55.544221 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532803    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 08:38:55.544221 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532806    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 08:38:55.544221 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532808    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 08:38:55.544221 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532811    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 08:38:55.544221 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532813    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 08:38:55.544221 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532816    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 08:38:55.544221 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532818    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 08:38:55.544221 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532822    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 08:38:55.544221 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532825    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 08:38:55.544684 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532828    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 08:38:55.544684 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532831    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 08:38:55.544684 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532833    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 08:38:55.544684 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532836    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 08:38:55.544684 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532838    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 08:38:55.544684 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532841    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 08:38:55.544684 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532844    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 08:38:55.544684 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532846    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 08:38:55.544684 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532849    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 08:38:55.544684 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532851    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 08:38:55.544684 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532854    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 08:38:55.544684 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532856    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 08:38:55.544684 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532860    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 08:38:55.544684 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532862    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 08:38:55.544684 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532865    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 08:38:55.544684 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532867    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 08:38:55.544684 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532869    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 08:38:55.544684 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532872    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 08:38:55.544684 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532874    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 08:38:55.544684 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532877    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 08:38:55.545178 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532879    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 08:38:55.545178 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532882    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 08:38:55.545178 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532885    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 08:38:55.545178 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532888    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 08:38:55.545178 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532891    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 08:38:55.545178 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532894    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 08:38:55.545178 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532896    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 08:38:55.545178 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532899    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 08:38:55.545178 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532901    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 08:38:55.545178 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532904    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 08:38:55.545178 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532906    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 08:38:55.545178 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532909    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 08:38:55.545178 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532911    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 08:38:55.545178 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532914    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 08:38:55.545178 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532916    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 08:38:55.545178 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532919    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 08:38:55.545178 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532922    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 08:38:55.545178 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532926    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 08:38:55.545178 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532929    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 08:38:55.545178 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532932    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 08:38:55.545660 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532934    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 08:38:55.545660 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532937    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 08:38:55.545660 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532939    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 08:38:55.545660 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532955    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 08:38:55.545660 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532958    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 08:38:55.545660 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532961    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 08:38:55.545660 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532964    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 08:38:55.545660 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532966    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 08:38:55.545660 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532969    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 08:38:55.545660 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532971    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 08:38:55.545660 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532974    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 08:38:55.545660 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532977    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 08:38:55.545660 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532979    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 08:38:55.545660 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532983    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 08:38:55.545660 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532985    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 08:38:55.545660 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532989    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 08:38:55.545660 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532992    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 08:38:55.545660 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532994    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 08:38:55.545660 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532997    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 08:38:55.546190 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.532999    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 08:38:55.546190 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.533002    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 08:38:55.546190 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.533004    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 08:38:55.546190 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.533007    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 08:38:55.546190 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.533010    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 08:38:55.546190 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.533012    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 08:38:55.546190 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.533015    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 08:38:55.546190 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.533017    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 08:38:55.546190 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.533020    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 08:38:55.546190 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.533022    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 08:38:55.546190 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.533025    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 08:38:55.546190 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.533027    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 08:38:55.546190 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.533030    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 08:38:55.546190 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.533033    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 08:38:55.546190 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.533035    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 08:38:55.546190 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.533038    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 08:38:55.546190 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.533040    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 08:38:55.546190 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.533044    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 08:38:55.546190 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.533046    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 08:38:55.546190 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.533049    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 08:38:55.546667 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.533052    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 08:38:55.546667 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.533054    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 08:38:55.546667 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.533057    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 08:38:55.546667 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.533060    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 08:38:55.546667 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.533062    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 08:38:55.546667 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.533065    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 08:38:55.546667 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.533067    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 08:38:55.546667 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.533071    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 08:38:55.546667 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.540137    2575 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 08:38:55.546667 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.540154    2575 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 08:38:55.546667 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.540171    2575 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 08:38:55.546667 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.540176    2575 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 08:38:55.546667 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540233    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 08:38:55.546667 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540238    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 08:38:55.546667 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540242    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 08:38:55.546667 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540245    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 08:38:55.546667 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540248    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 08:38:55.546667 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540251    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 08:38:55.546667 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540253    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 08:38:55.546667 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540256    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 08:38:55.547245 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540258    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 08:38:55.547245 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540261    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 08:38:55.547245 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540263    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 08:38:55.547245 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540265    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 08:38:55.547245 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540268    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 08:38:55.547245 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540271    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 08:38:55.547245 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540274    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 08:38:55.547245 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540276    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 08:38:55.547245 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540278    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 08:38:55.547245 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540281    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 08:38:55.547245 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540284    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 08:38:55.547245 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540286    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 08:38:55.547245 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540289    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 08:38:55.547245 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540291    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 08:38:55.547245 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540293    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 08:38:55.547245 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540296    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 08:38:55.547245 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540298    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 08:38:55.547245 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540301    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 08:38:55.547245 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540304    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 08:38:55.547245 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540306    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 08:38:55.547715 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540309    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 08:38:55.547715 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540312    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 08:38:55.547715 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540315    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 08:38:55.547715 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540318    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 08:38:55.547715 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540320    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 08:38:55.547715 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540323    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 08:38:55.547715 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540325    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 08:38:55.547715 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540328    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 08:38:55.547715 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540330    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 08:38:55.547715 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540333    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 08:38:55.547715 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540336    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 08:38:55.547715 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540338    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 08:38:55.547715 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540340    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 08:38:55.547715 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540343    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 08:38:55.547715 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540345    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 08:38:55.547715 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540347    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 08:38:55.547715 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540350    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 08:38:55.547715 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540352    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 08:38:55.547715 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540354    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 08:38:55.548174 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540357    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 08:38:55.548174 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540359    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 08:38:55.548174 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540362    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 08:38:55.548174 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540364    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 08:38:55.548174 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540366    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 08:38:55.548174 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540369    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 08:38:55.548174 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540371    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 08:38:55.548174 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540374    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 08:38:55.548174 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540376    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 08:38:55.548174 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540379    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 08:38:55.548174 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540381    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 08:38:55.548174 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540384    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 08:38:55.548174 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540386    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 08:38:55.548174 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540389    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 08:38:55.548174 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540391    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 08:38:55.548174 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540395    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 08:38:55.548174 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540407    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 08:38:55.548174 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540409    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 08:38:55.548174 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540412    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 08:38:55.548174 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540415    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 08:38:55.548675 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540417    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 08:38:55.548675 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540419    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 08:38:55.548675 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540422    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 08:38:55.548675 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540425    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 08:38:55.548675 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540427    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 08:38:55.548675 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540429    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 08:38:55.548675 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540432    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 08:38:55.548675 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540434    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 08:38:55.548675 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540437    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 08:38:55.548675 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540439    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 08:38:55.548675 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540442    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 08:38:55.548675 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540444    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 08:38:55.548675 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540446    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 08:38:55.548675 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540449    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 08:38:55.548675 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540451    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 08:38:55.548675 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540454    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 08:38:55.548675 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540456    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 08:38:55.548675 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540459    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 08:38:55.548675 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540461    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 08:38:55.548675 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540464    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 08:38:55.549184 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540466    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 08:38:55.549184 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540470    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 08:38:55.549184 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540475    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 08:38:55.549184 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540478    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 08:38:55.549184 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540481    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 08:38:55.549184 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540483    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 08:38:55.549184 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540486    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 08:38:55.549184 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540489    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 08:38:55.549184 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540492    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 08:38:55.549184 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540494    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 08:38:55.549184 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540497    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 08:38:55.549184 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540500    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 08:38:55.549184 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540502    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 08:38:55.549184 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540506    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 08:38:55.549184 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540510    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 08:38:55.549184 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540513    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 08:38:55.549184 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540515    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 08:38:55.549184 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540518    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 08:38:55.549603 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540520    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 08:38:55.549603 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.540547    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 08:38:55.549603 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540685    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 08:38:55.549603 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540689    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 08:38:55.549603 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540692    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 08:38:55.549603 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540695    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 08:38:55.549603 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540697    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 08:38:55.549603 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540700    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 08:38:55.549603 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540702    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 08:38:55.549603 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540704    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 08:38:55.549603 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540707    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 08:38:55.549603 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540710    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 08:38:55.549603 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540712    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 08:38:55.549603 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540715    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 08:38:55.549603 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540717    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 08:38:55.549603 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540720    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 08:38:55.550046 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540722    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 08:38:55.550046 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540725    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 08:38:55.550046 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540727    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 08:38:55.550046 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540730    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 08:38:55.550046 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540732    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 08:38:55.550046 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540735    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 08:38:55.550046 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540737    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 08:38:55.550046 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540740    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 08:38:55.550046 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540742    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 08:38:55.550046 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540745    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 08:38:55.550046 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540747    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 08:38:55.550046 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540749    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 08:38:55.550046 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540752    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 08:38:55.550046 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540754    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 08:38:55.550046 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540757    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 08:38:55.550046 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540759    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 08:38:55.550046 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540762    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 08:38:55.550046 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540764    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 08:38:55.550046 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540766    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 08:38:55.550046 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540769    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 08:38:55.550562 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540772    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 08:38:55.550562 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540774    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 08:38:55.550562 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540776    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 08:38:55.550562 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540779    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 08:38:55.550562 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540781    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 08:38:55.550562 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540783    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 08:38:55.550562 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540786    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 08:38:55.550562 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540788    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 08:38:55.550562 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540791    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 08:38:55.550562 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540795    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 08:38:55.550562 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540798    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 08:38:55.550562 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540801    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 08:38:55.550562 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540804    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 08:38:55.550562 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540806    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 08:38:55.550562 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540809    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 08:38:55.550562 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540813    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 08:38:55.550562 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540816    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 08:38:55.550562 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540818    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 08:38:55.550562 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540821    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 08:38:55.551019 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540823    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 08:38:55.551019 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540827    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 08:38:55.551019 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540830    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 08:38:55.551019 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540834    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 08:38:55.551019 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540837    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 08:38:55.551019 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540839    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 08:38:55.551019 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540842    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 08:38:55.551019 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540844    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 08:38:55.551019 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540847    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 08:38:55.551019 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540850    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 08:38:55.551019 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540852    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 08:38:55.551019 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540854    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 08:38:55.551019 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540857    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 08:38:55.551019 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540859    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 08:38:55.551019 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540861    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 08:38:55.551019 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540864    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 08:38:55.551019 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540866    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 08:38:55.551019 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540868    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 08:38:55.551019 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540871    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 08:38:55.551505 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540873    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 08:38:55.551505 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540876    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 08:38:55.551505 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540878    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 08:38:55.551505 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540880    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 08:38:55.551505 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540882    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 08:38:55.551505 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540885    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 08:38:55.551505 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540887    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 08:38:55.551505 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540890    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 08:38:55.551505 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540892    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 08:38:55.551505 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540895    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 08:38:55.551505 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540897    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 08:38:55.551505 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540917    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 08:38:55.551505 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540921    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 08:38:55.551505 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540924    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 08:38:55.551505 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540926    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 08:38:55.551505 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540930    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 08:38:55.551505 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540932    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 08:38:55.551505 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540935    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 08:38:55.551505 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540937    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 08:38:55.551505 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540940    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 08:38:55.551997 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540960    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 08:38:55.551997 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540963    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 08:38:55.551997 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540966    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 08:38:55.551997 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540968    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 08:38:55.551997 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540971    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 08:38:55.551997 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540973    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 08:38:55.551997 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540976    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 08:38:55.551997 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540979    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 08:38:55.551997 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540981    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 08:38:55.551997 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540984    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 08:38:55.551997 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540986    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 08:38:55.551997 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540988    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 08:38:55.551997 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540991    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 08:38:55.551997 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.540994    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 08:38:55.551997 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.541017    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 08:38:55.551997 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541050    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 08:38:55.552387 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541055    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 08:38:55.552387 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541058    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 08:38:55.552387 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541061    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 08:38:55.552387 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541064    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 08:38:55.552387 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541067    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 08:38:55.552387 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541070    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 08:38:55.552387 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541073    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 08:38:55.552387 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541075    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 08:38:55.552387 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541078    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 08:38:55.552387 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541080    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 08:38:55.552387 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541083    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 08:38:55.552387 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541085    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 08:38:55.552387 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541088    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 08:38:55.552387 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541091    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 08:38:55.552387 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541094    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 08:38:55.552387 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541096    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 08:38:55.552387 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541099    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 08:38:55.552387 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541101    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 08:38:55.552387 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541103    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 08:38:55.552387 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541106    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 08:38:55.552894 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541108    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 08:38:55.552894 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541110    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 08:38:55.552894 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541113    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 08:38:55.552894 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541115    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 08:38:55.552894 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541117    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 08:38:55.552894 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541120    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 08:38:55.552894 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541123    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 08:38:55.552894 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541125    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 08:38:55.552894 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541127    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 08:38:55.552894 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541130    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 08:38:55.552894 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541132    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 08:38:55.552894 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541135    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 08:38:55.552894 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541137    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 08:38:55.552894 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541139    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 08:38:55.552894 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541142    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 08:38:55.552894 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541144    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 08:38:55.552894 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541146    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 08:38:55.552894 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541149    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 08:38:55.552894 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541151    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 08:38:55.552894 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541153    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 08:38:55.553401 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541156    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 08:38:55.553401 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541158    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 08:38:55.553401 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541161    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 08:38:55.553401 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541163    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 08:38:55.553401 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541165    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 08:38:55.553401 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541171    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 08:38:55.553401 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541174    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 08:38:55.553401 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541176    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 08:38:55.553401 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541179    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 08:38:55.553401 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541182    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 08:38:55.553401 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541184    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 08:38:55.553401 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541186    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 08:38:55.553401 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541189    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 08:38:55.553401 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541191    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 08:38:55.553401 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541194    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 08:38:55.553401 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541196    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 08:38:55.553401 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541199    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 08:38:55.553401 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541202    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 08:38:55.553401 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541204    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 08:38:55.553401 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541206    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 08:38:55.553889 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541209    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 08:38:55.553889 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541211    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 08:38:55.553889 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541214    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 08:38:55.553889 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541216    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 08:38:55.553889 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541219    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 08:38:55.553889 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541221    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 08:38:55.553889 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541223    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 08:38:55.553889 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541226    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 08:38:55.553889 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541228    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 08:38:55.553889 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541231    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 08:38:55.553889 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541233    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 08:38:55.553889 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541235    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 08:38:55.553889 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541238    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 08:38:55.553889 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541240    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 08:38:55.553889 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541242    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 08:38:55.553889 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541245    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 08:38:55.553889 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541247    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 08:38:55.553889 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541251    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 08:38:55.553889 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541255    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 08:38:55.553889 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541257    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 08:38:55.554415 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541259    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 08:38:55.554415 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541262    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 08:38:55.554415 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541264    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 08:38:55.554415 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541267    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 08:38:55.554415 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541269    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 08:38:55.554415 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541272    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 08:38:55.554415 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541274    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 08:38:55.554415 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541277    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 08:38:55.554415 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541279    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 08:38:55.554415 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541282    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 08:38:55.554415 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541284    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 08:38:55.554415 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541288    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 08:38:55.554415 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541291    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 08:38:55.554415 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541294    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 08:38:55.554415 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541296    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 08:38:55.554415 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541299    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 08:38:55.554415 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541301    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 08:38:55.554415 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541304    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 08:38:55.554415 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541306    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 08:38:55.554845 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541311    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 08:38:55.554845 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541313    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 08:38:55.554845 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541316    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 08:38:55.554845 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541318    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 08:38:55.554845 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541323    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 08:38:55.554845 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:55.541326    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 08:38:55.554845 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.541396    2575 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 08:38:55.554845 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.547228    2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 08:38:55.554845 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.548249    2575 server.go:1014] "Starting client certificate rotation"
Nov 03 08:38:55.554845 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.548340    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 08:38:55.554845 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.548383    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 08:38:55.554845 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.550418    2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 08:38:55.554845 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.554285    2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 08:38:55.563634 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.563617    2575 log.go:25] "Validated CRI v1 runtime API"
Nov 03 08:38:55.567262 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.567246    2575 log.go:25] "Validated CRI v1 image API"
Nov 03 08:38:55.568121 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.568102    2575 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 08:38:55.575140 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.575121    2575 fs.go:135] Filesystem UUIDs: map[4e387ece-1ded-4257-a10a-c694e7b98d46:/dev/nvme0n1p3 669aaf52-edbc-4c40-9a6a-bb3102824100:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Nov 03 08:38:55.575188 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.575141    2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 08:38:55.579635 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.579528    2575 manager.go:217] Machine: {Timestamp:2026-11-03 08:38:55.57829611 +0000 UTC m=+0.351916160 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3108485 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec297d172f32aa054a5dac938116ab5a SystemUUID:ec297d17-2f32-aa05-4a5d-ac938116ab5a BootID:0992e8ce-4269-4c76-bfa9-9413680324e8 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:12:f8:92:65:b1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:12:f8:92:65:b1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ea:43:23:91:bb:87 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 08:38:55.579635 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.579635    2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 08:38:55.579775 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.579736    2575 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 08:38:55.583345 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.583321    2575 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 08:38:55.583471 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.583347    2575 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-73.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 08:38:55.583511 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.583473    2575 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 08:38:55.583511 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.583481    2575 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 08:38:55.583511 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.583505    2575 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 08:38:55.584366 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.584356    2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 08:38:55.585599 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.585590    2575 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 08:38:55.589902 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.589890    2575 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 08:38:55.590031 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.590021    2575 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 08:38:55.590852 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.590843    2575 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 08:38:55.590903 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.590856    2575 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 08:38:55.590903 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.590872    2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 08:38:55.590903 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.590881    2575 kubelet.go:405] "Adding apiserver pod source"
Nov 03 08:38:55.590903 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.590895    2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 08:38:55.591225 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.591214    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 08:38:55.594934 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.594916    2575 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 08:38:55.602227 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.602212    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 08:38:55.603148 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.603134    2575 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 08:38:55.603186 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.603164    2575 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 08:38:55.603329 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.603320    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 08:38:55.603364 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.603335    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 08:38:55.603364 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.603342    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 08:38:55.603364 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.603347    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 08:38:55.603364 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.603353    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 08:38:55.603364 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.603358    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 08:38:55.603987 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.603978    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 08:38:55.604017 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.603987    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 08:38:55.604017 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.603994    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 08:38:55.604017 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.604000    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 08:38:55.604017 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.604008    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 08:38:55.604017 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.604017    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 08:38:55.605628 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.605619    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 08:38:55.605628 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.605628    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 08:38:55.609382 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.609366    2575 server.go:1263] "Started kubelet"
Nov 03 08:38:55.609531 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.609478    2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 08:38:55.609570 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.609555    2575 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 08:38:55.609777 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.609755    2575 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 08:38:55.610069 ip-10-0-139-73 systemd[1]: Started Kubernetes Kubelet.
Nov 03 08:38:55.611008 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.610991    2575 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 08:38:55.612469 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.612453    2575 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 08:38:55.617560 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:55.617529    2575 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 08:38:55.618487 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.618472    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 08:38:55.619729 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.619715    2575 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 08:38:55.619863 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:55.619825    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-73.us-west-1.compute.internal\" not found"
Nov 03 08:38:55.620528 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.620510    2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 08:38:55.620528 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.620514    2575 factory.go:55] Registering systemd factory
Nov 03 08:38:55.620681 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.620539    2575 factory.go:223] Registration of the systemd container factory successfully
Nov 03 08:38:55.620681 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.620515    2575 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 08:38:55.620681 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.620588    2575 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 08:38:55.620681 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.620662    2575 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 08:38:55.620681 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.620673    2575 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 08:38:55.620823 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.620741    2575 factory.go:153] Registering CRI-O factory
Nov 03 08:38:55.620823 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.620754    2575 factory.go:223] Registration of the crio container factory successfully
Nov 03 08:38:55.620960 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.620880    2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 08:38:55.621028 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.621006    2575 factory.go:103] Registering Raw factory
Nov 03 08:38:55.621028 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.621022    2575 manager.go:1199] Started watching for new ooms in manager
Nov 03 08:38:55.622097 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.622080    2575 manager.go:319] Starting recovery of all containers
Nov 03 08:38:55.632371 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.632353    2575 manager.go:324] Recovery completed
Nov 03 08:38:55.637558 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.637540    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 08:38:55.639038 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.639020    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 08:38:55.639111 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.639051    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 08:38:55.639111 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.639060    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 08:38:55.639556 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.639541    2575 cpu_manager.go:226] "Starting" policy="none"
Nov 03 08:38:55.639556 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.639551    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 08:38:55.639683 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.639554    2575 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 08:38:55.639683 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.639586    2575 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 08:38:55.640297 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.640280    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 08:38:55.640366 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.640308    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 08:38:55.640366 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.640317    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 08:38:55.641579 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.641566    2575 policy_none.go:50] "Start"
Nov 03 08:38:55.641642 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.641582    2575 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 08:38:55.641642 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.641593    2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 08:38:55.643506 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.643495    2575 policy_none.go:44] "Start"
Nov 03 08:38:55.682852 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.682822    2575 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 08:38:55.706572 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.687089    2575 manager.go:341] "Starting Device Plugin manager"
Nov 03 08:38:55.706572 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:55.687123    2575 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 08:38:55.706572 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.687134    2575 server.go:87] "Starting device plugin registration server"
Nov 03 08:38:55.706572 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.687383    2575 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 08:38:55.706572 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.687396    2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 08:38:55.706572 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.687635    2575 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 08:38:55.706572 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.687713    2575 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 08:38:55.706572 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.687719    2575 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 08:38:55.706572 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:55.688141    2575 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 08:38:55.706572 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:55.688186    2575 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-73.us-west-1.compute.internal\" not found"
Nov 03 08:38:55.706572 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.701055    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 08:38:55.706572 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.702325    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 08:38:55.706572 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.702345    2575 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 08:38:55.706572 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.702365    2575 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 08:38:55.706572 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:55.702401    2575 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 08:38:55.751989 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.751964    2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-73.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 08:38:55.751989 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:55.751962    2575 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-73.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 08:38:55.788482 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.788405    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 08:38:55.789315 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.789297    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 08:38:55.789366 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.789339    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 08:38:55.789366 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.789353    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 08:38:55.789450 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.789378    2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:38:55.801675 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:55.800765    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-73.us-west-1.compute.internal.18e47c38f1573f93  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-73.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-73.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 08:38:55.609339795 +0000 UTC m=+0.382959840,LastTimestamp:2026-11-03 08:38:55.609339795 +0000 UTC m=+0.382959840,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-73.us-west-1.compute.internal,}"
Nov 03 08:38:55.803617 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.803599    2575 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-73.us-west-1.compute.internal"]
Nov 03 08:38:55.803671 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.803652    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 08:38:55.804481 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.804467    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 08:38:55.804533 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.804494    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 08:38:55.804533 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.804504    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 08:38:55.806625 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.806614    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 08:38:55.806759 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.806744    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:38:55.806799 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.806774    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 08:38:55.807314 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.807295    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 08:38:55.807416 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.807325    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 08:38:55.807416 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.807341    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 08:38:55.807416 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.807347    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 08:38:55.807416 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.807364    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 08:38:55.807416 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.807384    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 08:38:55.809533 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.809517    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:38:55.809629 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.809543    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 08:38:55.810278 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.810258    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 08:38:55.810342 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.810295    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 08:38:55.810342 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.810310    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 08:38:55.823167 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:55.823135    2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-73.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 03 08:38:55.843442 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:55.843427    2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-73.us-west-1.compute.internal\" not found" node="ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:38:55.846012 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:55.845997    2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-73.us-west-1.compute.internal\" not found" node="ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:38:55.859054 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.859039    2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 08:38:55.859134 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.859076    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j6j2t"
Nov 03 08:38:55.906615 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:55.906526    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-73.us-west-1.compute.internal.18e47c38f31c660d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-73.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-73.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-73.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 08:38:55.639037453 +0000 UTC m=+0.412657497,LastTimestamp:2026-11-03 08:38:55.639037453 +0000 UTC m=+0.412657497,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-73.us-west-1.compute.internal,}"
Nov 03 08:38:55.906719 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:55.906650    2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 03 08:38:55.906719 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:55.906694    2575 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 03 08:38:55.921428 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.921409    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bda1773f2a5825c17c394ccc2e58c1ce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal\" (UID: \"bda1773f2a5825c17c394ccc2e58c1ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:38:55.921479 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.921433    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bda1773f2a5825c17c394ccc2e58c1ce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal\" (UID: \"bda1773f2a5825c17c394ccc2e58c1ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:38:55.921479 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:55.921450    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0f8947715126d508f609643819d8b2a7-config\") pod \"kube-apiserver-proxy-ip-10-0-139-73.us-west-1.compute.internal\" (UID: \"0f8947715126d508f609643819d8b2a7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:38:55.955782 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:55.955757    2575 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Nov 03 08:38:55.982536 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:55.982516    2575 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-139-73.us-west-1.compute.internal\" not found" node="ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:38:56.021826 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:56.021806    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bda1773f2a5825c17c394ccc2e58c1ce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal\" (UID: \"bda1773f2a5825c17c394ccc2e58c1ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:38:56.021899 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:56.021831    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bda1773f2a5825c17c394ccc2e58c1ce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal\" (UID: \"bda1773f2a5825c17c394ccc2e58c1ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:38:56.021899 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:56.021849    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0f8947715126d508f609643819d8b2a7-config\") pod \"kube-apiserver-proxy-ip-10-0-139-73.us-west-1.compute.internal\" (UID: \"0f8947715126d508f609643819d8b2a7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:38:56.021899 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:56.021882    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0f8947715126d508f609643819d8b2a7-config\") pod \"kube-apiserver-proxy-ip-10-0-139-73.us-west-1.compute.internal\" (UID: \"0f8947715126d508f609643819d8b2a7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:38:56.021899 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:56.021887    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bda1773f2a5825c17c394ccc2e58c1ce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal\" (UID: \"bda1773f2a5825c17c394ccc2e58c1ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:38:56.022054 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:56.021904    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bda1773f2a5825c17c394ccc2e58c1ce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal\" (UID: \"bda1773f2a5825c17c394ccc2e58c1ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:38:56.057578 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:56.057528    2575 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-73.us-west-1.compute.internal\" not found" node="ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:38:56.148012 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:56.147989    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:38:56.151706 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:56.151685    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:38:56.183181 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:56.183165    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 08:38:56.183984 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:56.183968    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 08:38:56.184069 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:56.184003    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 08:38:56.184069 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:56.184020    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 08:38:56.184069 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:56.184053    2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:38:56.239910 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:56.239886    2575 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:38:56.240029 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:56.239911    2575 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-73.us-west-1.compute.internal\": node \"ip-10-0-139-73.us-west-1.compute.internal\" not found"
Nov 03 08:38:56.357414 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:56.357350    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-73.us-west-1.compute.internal\" not found"
Nov 03 08:38:56.458168 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:56.458145    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-73.us-west-1.compute.internal\" not found"
Nov 03 08:38:56.547773 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:56.547751    2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 08:38:56.548215 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:56.547888    2575 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://aaf99b2950bda4e3eb23748958ae1964-ce6ebf1c3713135a.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-139-73.us-west-1.compute.internal.18e47c38f31ccf7c\": read tcp 10.0.139.73:53300->18.226.136.203:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-139-73.us-west-1.compute.internal.18e47c38f31ccf7c  default   4997 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-73.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-73.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-73.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 08:38:55 +0000 UTC,LastTimestamp:2026-11-03 08:38:55.789357916 +0000 UTC m=+0.562977961,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-73.us-west-1.compute.internal,}"
Nov 03 08:38:56.559061 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:56.559039    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-73.us-west-1.compute.internal\" not found"
Nov 03 08:38:56.659841 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:56.659785    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-73.us-west-1.compute.internal\" not found"
Nov 03 08:38:56.760843 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:56.760817    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-73.us-west-1.compute.internal\" not found"
Nov 03 08:38:56.861778 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:56.861757    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-73.us-west-1.compute.internal\" not found"
Nov 03 08:38:56.864916 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:56.864893    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 08:33:55 +0000 UTC" deadline="2028-07-24 03:17:26.280142326 +0000 UTC"
Nov 03 08:38:56.864987 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:56.864917    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15090h38m29.41522863s"
Nov 03 08:38:56.877047 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:56.877034    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 08:38:56.877895 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:56.877880    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 08:38:56.877979 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:56.877907    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 08:38:56.877979 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:56.877926    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 08:38:56.910568 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:56.910524    2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 08:38:56.962025 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:56.962001    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-73.us-west-1.compute.internal\" not found"
Nov 03 08:38:57.063229 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:57.063058    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-73.us-west-1.compute.internal\" not found"
Nov 03 08:38:57.164053 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:57.163985    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-73.us-west-1.compute.internal\" not found"
Nov 03 08:38:57.165885 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.165855    2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 08:38:57.173490 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.173472    2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 08:38:57.220485 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.220468    2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:38:57.225173 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:57.225141    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f8947715126d508f609643819d8b2a7.slice/crio-b580705cc718f166d5784f6f78ecc317a4b988d9a0cae191724a15bd2df10c5b WatchSource:0}: Error finding container b580705cc718f166d5784f6f78ecc317a4b988d9a0cae191724a15bd2df10c5b: Status 404 returned error can't find the container with id b580705cc718f166d5784f6f78ecc317a4b988d9a0cae191724a15bd2df10c5b
Nov 03 08:38:57.225623 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:57.225605    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbda1773f2a5825c17c394ccc2e58c1ce.slice/crio-3cacd773f4794a07bd9127962388165da3304c7adda2ab3b7548a9a22f49ac05 WatchSource:0}: Error finding container 3cacd773f4794a07bd9127962388165da3304c7adda2ab3b7548a9a22f49ac05: Status 404 returned error can't find the container with id 3cacd773f4794a07bd9127962388165da3304c7adda2ab3b7548a9a22f49ac05
Nov 03 08:38:57.229699 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.229686    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 08:38:57.279164 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.279146    2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 08:38:57.280897 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.280886    2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:38:57.335210 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.335191    2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 08:38:57.521203 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.521135    2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 08:38:57.592121 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.592101    2575 apiserver.go:52] "Watching apiserver"
Nov 03 08:38:57.620281 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.620262    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 08:38:57.688310 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.688284    2575 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 08:38:57.688599 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.688581    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-pflts","kube-system/konnectivity-agent-bq5p4","kube-system/kube-apiserver-proxy-ip-10-0-139-73.us-west-1.compute.internal","openshift-network-diagnostics/network-check-target-clx84","openshift-network-operator/iptables-alerter-689v8","openshift-ovn-kubernetes/ovnkube-node-bt4zt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs","openshift-cluster-node-tuning-operator/tuned-jpll8","openshift-dns/node-resolver-hgbb5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-wsrmr","kube-system/global-pull-secret-syncer-x9jpw","openshift-image-registry/node-ca-5drgw","openshift-multus/multus-kzcqg"]
Nov 03 08:38:57.692129 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.692112    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:38:57.693998 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.693981    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5drgw"
Nov 03 08:38:57.695906 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.695891    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.697884 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.697863    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-clx84"
Nov 03 08:38:57.697992 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:57.697957    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-clx84" podUID="f6ca3dc3-231e-4a8d-a16c-7384c225110a"
Nov 03 08:38:57.699749 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.699734    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-689v8"
Nov 03 08:38:57.702438 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.702419    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.704389 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.704375    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hgbb5"
Nov 03 08:38:57.706426 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.706412    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pflts"
Nov 03 08:38:57.706503 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:57.706466    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pflts" podUID="7c26dfa4-e47e-43a1-94a2-816d22fb7f3a"
Nov 03 08:38:57.708582 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.708478    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bq5p4"
Nov 03 08:38:57.710587 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.710573    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.712666 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.712652    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:38:57.715045 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.715030    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:38:57.715130 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:57.715096    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9jpw" podUID="906c75b6-325c-4e89-8a10-e4e355884133"
Nov 03 08:38:57.717053 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.717023    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal" event={"ID":"bda1773f2a5825c17c394ccc2e58c1ce","Type":"ContainerStarted","Data":"3cacd773f4794a07bd9127962388165da3304c7adda2ab3b7548a9a22f49ac05"}
Nov 03 08:38:57.717107 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.717064    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-73.us-west-1.compute.internal" event={"ID":"0f8947715126d508f609643819d8b2a7","Type":"ContainerStarted","Data":"b580705cc718f166d5784f6f78ecc317a4b988d9a0cae191724a15bd2df10c5b"}
Nov 03 08:38:57.721326 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.721314    2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 08:38:57.731140 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.731084    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7e02a4de-6df2-4930-8360-4c5e09897472-socket-dir\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:38:57.731222 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.731141    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/906c75b6-325c-4e89-8a10-e4e355884133-dbus\") pod \"global-pull-secret-syncer-x9jpw\" (UID: \"906c75b6-325c-4e89-8a10-e4e355884133\") " pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:38:57.731222 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.731176    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-var-lib-openvswitch\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.731222 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.731205    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0a5d9c55-0799-4ae1-a902-dc06f813907c-ovn-node-metrics-cert\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.731400 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.731240    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-run-ovn-kubernetes\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.731400 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.731271    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7e02a4de-6df2-4930-8360-4c5e09897472-registration-dir\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:38:57.731400 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.731298    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6b529371-cf9d-40e4-a2c5-55d69ab9a08d-iptables-alerter-script\") pod \"iptables-alerter-689v8\" (UID: \"6b529371-cf9d-40e4-a2c5-55d69ab9a08d\") " pod="openshift-network-operator/iptables-alerter-689v8"
Nov 03 08:38:57.731400 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.731386    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/110651f3-9a70-43bf-888d-0813682bc891-serviceca\") pod \"node-ca-5drgw\" (UID: \"110651f3-9a70-43bf-888d-0813682bc891\") " pod="openshift-image-registry/node-ca-5drgw"
Nov 03 08:38:57.731570 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.731423    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-cnibin\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.731570 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.731453    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-hostroot\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.731570 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.731472    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-etc-kubernetes\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.731570 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.731531    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/906c75b6-325c-4e89-8a10-e4e355884133-kubelet-config\") pod \"global-pull-secret-syncer-x9jpw\" (UID: \"906c75b6-325c-4e89-8a10-e4e355884133\") " pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:38:57.731722 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.731577    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/110651f3-9a70-43bf-888d-0813682bc891-host\") pod \"node-ca-5drgw\" (UID: \"110651f3-9a70-43bf-888d-0813682bc891\") " pod="openshift-image-registry/node-ca-5drgw"
Nov 03 08:38:57.731722 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.731612    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-etc-openvswitch\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.731848 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.731804    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-run-ovn\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.731848 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.731840    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-log-socket\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.731971 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.731877    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-modprobe-d\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.731971 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.731905    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-systemd\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.731971 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.731959    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/801881d6-f217-4c80-a190-af953df3af04-tmp-dir\") pod \"node-resolver-hgbb5\" (UID: \"801881d6-f217-4c80-a190-af953df3af04\") " pod="openshift-dns/node-resolver-hgbb5"
Nov 03 08:38:57.732131 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.732116    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bg9b\" (UniqueName: \"kubernetes.io/projected/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-kube-api-access-8bg9b\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.732175 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.732152    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-systemd-units\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.732220 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.732181    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-cni-netd\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.732220 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.732207    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-tuned\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.732301 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.732227    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1649ad9-921c-49d9-baa5-c51f44175b8c-tmp\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.732301 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.732258    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wsrmr\" (UID: \"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9\") " pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:38:57.732301 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.732288    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wsrmr\" (UID: \"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9\") " pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:38:57.732423 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.732315    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0a5d9c55-0799-4ae1-a902-dc06f813907c-env-overrides\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.732423 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.732343    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wsrmr\" (UID: \"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9\") " pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:38:57.732423 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.732366    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-system-cni-dir\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.732423 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.732392    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e1386760-3800-418f-bb1b-20e19880620b-konnectivity-ca\") pod \"konnectivity-agent-bq5p4\" (UID: \"e1386760-3800-418f-bb1b-20e19880620b\") " pod="kube-system/konnectivity-agent-bq5p4"
Nov 03 08:38:57.732423 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.732417    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-host-var-lib-kubelet\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.732621 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.732504    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnnxt\" (UniqueName: \"kubernetes.io/projected/d1649ad9-921c-49d9-baa5-c51f44175b8c-kube-api-access-xnnxt\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.732621 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.732531    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-cnibin\") pod \"multus-additional-cni-plugins-wsrmr\" (UID: \"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9\") " pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:38:57.732621 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.732554    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/906c75b6-325c-4e89-8a10-e4e355884133-original-pull-secret\") pod \"global-pull-secret-syncer-x9jpw\" (UID: \"906c75b6-325c-4e89-8a10-e4e355884133\") " pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:38:57.732621 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.732580    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-cni-binary-copy\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.732621 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.732611    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-host-run-multus-certs\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.732813 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.732637    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-run-systemd\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.732813 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.732666    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.732813 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.732694    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4vs4\" (UniqueName: \"kubernetes.io/projected/0a5d9c55-0799-4ae1-a902-dc06f813907c-kube-api-access-s4vs4\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.732813 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.732717    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-run\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.732813 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.732739    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-cni-bin\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.732813 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.732764    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7e02a4de-6df2-4930-8360-4c5e09897472-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:38:57.732813 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.732795    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvsbx\" (UniqueName: \"kubernetes.io/projected/7e02a4de-6df2-4930-8360-4c5e09897472-kube-api-access-jvsbx\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:38:57.733121 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.732821    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e1386760-3800-418f-bb1b-20e19880620b-agent-certs\") pod \"konnectivity-agent-bq5p4\" (UID: \"e1386760-3800-418f-bb1b-20e19880620b\") " pod="kube-system/konnectivity-agent-bq5p4"
Nov 03 08:38:57.733121 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.732843    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-sysctl-conf\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.733121 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.732879    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rxtf\" (UniqueName: \"kubernetes.io/projected/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-kube-api-access-7rxtf\") pod \"multus-additional-cni-plugins-wsrmr\" (UID: \"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9\") " pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:38:57.733121 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.732903    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-slash\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.733121 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.732927    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7e02a4de-6df2-4930-8360-4c5e09897472-device-dir\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:38:57.733121 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.732979    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7e02a4de-6df2-4930-8360-4c5e09897472-etc-selinux\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:38:57.733121 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.732999    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/801881d6-f217-4c80-a190-af953df3af04-hosts-file\") pod \"node-resolver-hgbb5\" (UID: \"801881d6-f217-4c80-a190-af953df3af04\") " pod="openshift-dns/node-resolver-hgbb5"
Nov 03 08:38:57.733121 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.733023    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c26dfa4-e47e-43a1-94a2-816d22fb7f3a-metrics-certs\") pod \"network-metrics-daemon-pflts\" (UID: \"7c26dfa4-e47e-43a1-94a2-816d22fb7f3a\") " pod="openshift-multus/network-metrics-daemon-pflts"
Nov 03 08:38:57.733121 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.733049    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99jxr\" (UniqueName: \"kubernetes.io/projected/7c26dfa4-e47e-43a1-94a2-816d22fb7f3a-kube-api-access-99jxr\") pod \"network-metrics-daemon-pflts\" (UID: \"7c26dfa4-e47e-43a1-94a2-816d22fb7f3a\") " pod="openshift-multus/network-metrics-daemon-pflts"
Nov 03 08:38:57.733121 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.733074    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0a5d9c55-0799-4ae1-a902-dc06f813907c-ovnkube-script-lib\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.733121 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.733098    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-var-lib-kubelet\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.733574 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.733230    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgfkn\" (UniqueName: \"kubernetes.io/projected/6b529371-cf9d-40e4-a2c5-55d69ab9a08d-kube-api-access-sgfkn\") pod \"iptables-alerter-689v8\" (UID: \"6b529371-cf9d-40e4-a2c5-55d69ab9a08d\") " pod="openshift-network-operator/iptables-alerter-689v8"
Nov 03 08:38:57.733574 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.733294    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njvj8\" (UniqueName: \"kubernetes.io/projected/801881d6-f217-4c80-a190-af953df3af04-kube-api-access-njvj8\") pod \"node-resolver-hgbb5\" (UID: \"801881d6-f217-4c80-a190-af953df3af04\") " pod="openshift-dns/node-resolver-hgbb5"
Nov 03 08:38:57.733574 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.733410    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-run-openvswitch\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.733574 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.733460    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-multus-daemon-config\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.733727 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.733580    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-sysconfig\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.733727 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.733651    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-lib-modules\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.733727 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.733693    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-system-cni-dir\") pod \"multus-additional-cni-plugins-wsrmr\" (UID: \"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9\") " pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:38:57.733879 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.733734    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-multus-cni-dir\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.733879 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.733782    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-kubernetes\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.733879 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.733875    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-multus-socket-dir-parent\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.734027 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.733919    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-node-log\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.734290 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.734268    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-sysctl-d\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.734348 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.734320    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-sys\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.734402 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.734381    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-host\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.734460 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.734442    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7e02a4de-6df2-4930-8360-4c5e09897472-sys-fs\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:38:57.734519 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.734503    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpfx2\" (UniqueName: \"kubernetes.io/projected/110651f3-9a70-43bf-888d-0813682bc891-kube-api-access-hpfx2\") pod \"node-ca-5drgw\" (UID: \"110651f3-9a70-43bf-888d-0813682bc891\") " pod="openshift-image-registry/node-ca-5drgw"
Nov 03 08:38:57.734551 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.734536    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-host-run-k8s-cni-cncf-io\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.734594 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.734570    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-host-run-netns\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.734627 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.734600    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-host-var-lib-cni-multus\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.734627 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.734618    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-multus-conf-dir\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.734701 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.734663    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-kubelet\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.734741 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.734708    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-run-netns\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.734789 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.734739    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b529371-cf9d-40e4-a2c5-55d69ab9a08d-host-slash\") pod \"iptables-alerter-689v8\" (UID: \"6b529371-cf9d-40e4-a2c5-55d69ab9a08d\") " pod="openshift-network-operator/iptables-alerter-689v8"
Nov 03 08:38:57.734832 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.734787    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-host-var-lib-cni-bin\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.734869 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.734827    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcm5z\" (UniqueName: \"kubernetes.io/projected/f6ca3dc3-231e-4a8d-a16c-7384c225110a-kube-api-access-fcm5z\") pod \"network-check-target-clx84\" (UID: \"f6ca3dc3-231e-4a8d-a16c-7384c225110a\") " pod="openshift-network-diagnostics/network-check-target-clx84"
Nov 03 08:38:57.734913 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.734891    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0a5d9c55-0799-4ae1-a902-dc06f813907c-ovnkube-config\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.734968 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.734922    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-cni-binary-copy\") pod \"multus-additional-cni-plugins-wsrmr\" (UID: \"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9\") " pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:38:57.741920 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.741902    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 08:38:57.741920 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.741914    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 08:38:57.742072 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.741976    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v9lhl\""
Nov 03 08:38:57.742072 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.741988    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 08:38:57.742344 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.742326    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 08:38:57.742437 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.742384    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 08:38:57.742437 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.742423    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jjcxd\""
Nov 03 08:38:57.742517 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.742455    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 08:38:57.744094 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.744080    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 08:38:57.744195 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.744182    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jnk2x\""
Nov 03 08:38:57.744251 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.744238    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 08:38:57.744282 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.744253    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 08:38:57.744282 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.744258    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 08:38:57.748042 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.748028    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 08:38:57.748111 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.748082    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 08:38:57.748149 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.748120    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 08:38:57.748149 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.748134    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sntnv\""
Nov 03 08:38:57.750704 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.750693    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 08:38:57.750846 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.750837    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 08:38:57.750846 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.750875    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 08:38:57.751094 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.751082    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 08:38:57.776413 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.776371    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 08:38:57.776474 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.776442    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 08:38:57.776889 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.776867    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hvj2z\""
Nov 03 08:38:57.776984 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.776893    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 08:38:57.776984 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.776900    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 08:38:57.777048 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.776991    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 08:38:57.777048 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.777003    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 08:38:57.777048 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.777022    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 08:38:57.777048 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.777035    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rnxpp\""
Nov 03 08:38:57.777158 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.777130    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cwzpz\""
Nov 03 08:38:57.777158 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.777135    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 08:38:57.777221 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.777165    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8lmql\""
Nov 03 08:38:57.777221 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.777172    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tps59\""
Nov 03 08:38:57.781751 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.781735    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 08:38:57.781751 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.781745    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 08:38:57.835295 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.835279    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/906c75b6-325c-4e89-8a10-e4e355884133-original-pull-secret\") pod \"global-pull-secret-syncer-x9jpw\" (UID: \"906c75b6-325c-4e89-8a10-e4e355884133\") " pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:38:57.835398 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.835301    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-cni-binary-copy\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.835398 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.835315    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-host-run-multus-certs\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.835398 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.835331    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-run-systemd\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.835398 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.835345    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.835398 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.835359    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4vs4\" (UniqueName: \"kubernetes.io/projected/0a5d9c55-0799-4ae1-a902-dc06f813907c-kube-api-access-s4vs4\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.835398 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.835377    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-run\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.835398 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.835393    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-cni-bin\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.835668 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.835427    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.835668 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.835461    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-cni-bin\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.835668 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.835464    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-run-systemd\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.835668 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.835428    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-host-run-multus-certs\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.835668 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:57.835492    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 08:38:57.835668 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.835414    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7e02a4de-6df2-4930-8360-4c5e09897472-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:38:57.835668 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.835537    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-run\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.835668 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:57.835557    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/906c75b6-325c-4e89-8a10-e4e355884133-original-pull-secret podName:906c75b6-325c-4e89-8a10-e4e355884133 nodeName:}" failed. No retries permitted until 2026-11-03 08:38:58.335519925 +0000 UTC m=+3.109139954 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/906c75b6-325c-4e89-8a10-e4e355884133-original-pull-secret") pod "global-pull-secret-syncer-x9jpw" (UID: "906c75b6-325c-4e89-8a10-e4e355884133") : object "kube-system"/"original-pull-secret" not registered
Nov 03 08:38:57.835668 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.835576    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvsbx\" (UniqueName: \"kubernetes.io/projected/7e02a4de-6df2-4930-8360-4c5e09897472-kube-api-access-jvsbx\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:38:57.835668 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.835580    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7e02a4de-6df2-4930-8360-4c5e09897472-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:38:57.835668 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.835613    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e1386760-3800-418f-bb1b-20e19880620b-agent-certs\") pod \"konnectivity-agent-bq5p4\" (UID: \"e1386760-3800-418f-bb1b-20e19880620b\") " pod="kube-system/konnectivity-agent-bq5p4"
Nov 03 08:38:57.835668 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.835636    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-sysctl-conf\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.835668 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.835659    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rxtf\" (UniqueName: \"kubernetes.io/projected/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-kube-api-access-7rxtf\") pod \"multus-additional-cni-plugins-wsrmr\" (UID: \"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9\") " pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:38:57.836236 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.835682    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-slash\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.836236 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.835701    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7e02a4de-6df2-4930-8360-4c5e09897472-device-dir\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:38:57.836236 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.835719    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7e02a4de-6df2-4930-8360-4c5e09897472-etc-selinux\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:38:57.836236 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.835737    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/801881d6-f217-4c80-a190-af953df3af04-hosts-file\") pod \"node-resolver-hgbb5\" (UID: \"801881d6-f217-4c80-a190-af953df3af04\") " pod="openshift-dns/node-resolver-hgbb5"
Nov 03 08:38:57.836236 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.835739    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-slash\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.836236 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.835757    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c26dfa4-e47e-43a1-94a2-816d22fb7f3a-metrics-certs\") pod \"network-metrics-daemon-pflts\" (UID: \"7c26dfa4-e47e-43a1-94a2-816d22fb7f3a\") " pod="openshift-multus/network-metrics-daemon-pflts"
Nov 03 08:38:57.836236 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.835807    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-sysctl-conf\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.836236 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.835840    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-cni-binary-copy\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.836236 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.835913    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7e02a4de-6df2-4930-8360-4c5e09897472-device-dir\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:38:57.836236 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:57.835958    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 08:38:57.836236 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.835975    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99jxr\" (UniqueName: \"kubernetes.io/projected/7c26dfa4-e47e-43a1-94a2-816d22fb7f3a-kube-api-access-99jxr\") pod \"network-metrics-daemon-pflts\" (UID: \"7c26dfa4-e47e-43a1-94a2-816d22fb7f3a\") " pod="openshift-multus/network-metrics-daemon-pflts"
Nov 03 08:38:57.836236 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.835973    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/801881d6-f217-4c80-a190-af953df3af04-hosts-file\") pod \"node-resolver-hgbb5\" (UID: \"801881d6-f217-4c80-a190-af953df3af04\") " pod="openshift-dns/node-resolver-hgbb5"
Nov 03 08:38:57.836236 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.835981    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7e02a4de-6df2-4930-8360-4c5e09897472-etc-selinux\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:38:57.836236 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:57.836013    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c26dfa4-e47e-43a1-94a2-816d22fb7f3a-metrics-certs podName:7c26dfa4-e47e-43a1-94a2-816d22fb7f3a nodeName:}" failed. No retries permitted until 2026-11-03 08:38:58.335998248 +0000 UTC m=+3.109618277 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c26dfa4-e47e-43a1-94a2-816d22fb7f3a-metrics-certs") pod "network-metrics-daemon-pflts" (UID: "7c26dfa4-e47e-43a1-94a2-816d22fb7f3a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 08:38:57.836236 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836035    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0a5d9c55-0799-4ae1-a902-dc06f813907c-ovnkube-script-lib\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.836236 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836036    2575 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 08:38:57.836236 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836124    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-var-lib-kubelet\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.836741 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836140    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgfkn\" (UniqueName: \"kubernetes.io/projected/6b529371-cf9d-40e4-a2c5-55d69ab9a08d-kube-api-access-sgfkn\") pod \"iptables-alerter-689v8\" (UID: \"6b529371-cf9d-40e4-a2c5-55d69ab9a08d\") " pod="openshift-network-operator/iptables-alerter-689v8"
Nov 03 08:38:57.836741 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836162    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njvj8\" (UniqueName: \"kubernetes.io/projected/801881d6-f217-4c80-a190-af953df3af04-kube-api-access-njvj8\") pod \"node-resolver-hgbb5\" (UID: \"801881d6-f217-4c80-a190-af953df3af04\") " pod="openshift-dns/node-resolver-hgbb5"
Nov 03 08:38:57.836741 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836183    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-run-openvswitch\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.836741 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836189    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-var-lib-kubelet\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.836741 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836207    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-multus-daemon-config\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.836741 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836264    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-sysconfig\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.836741 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836293    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-run-openvswitch\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.836741 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836323    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-sysconfig\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.836741 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836326    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-lib-modules\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.836741 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836375    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-system-cni-dir\") pod \"multus-additional-cni-plugins-wsrmr\" (UID: \"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9\") " pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:38:57.836741 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836396    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-multus-cni-dir\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.836741 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836399    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-lib-modules\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.836741 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836410    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-kubernetes\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.836741 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836424    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-multus-socket-dir-parent\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.836741 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836444    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-node-log\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.836741 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836464    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-sysctl-d\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.836741 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836483    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-sys\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.836741 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836502    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-host\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.837431 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836517    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-kubernetes\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.837431 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836524    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7e02a4de-6df2-4930-8360-4c5e09897472-sys-fs\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:38:57.837431 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836505    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-system-cni-dir\") pod \"multus-additional-cni-plugins-wsrmr\" (UID: \"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9\") " pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:38:57.837431 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836562    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpfx2\" (UniqueName: \"kubernetes.io/projected/110651f3-9a70-43bf-888d-0813682bc891-kube-api-access-hpfx2\") pod \"node-ca-5drgw\" (UID: \"110651f3-9a70-43bf-888d-0813682bc891\") " pod="openshift-image-registry/node-ca-5drgw"
Nov 03 08:38:57.837431 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836566    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-node-log\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.837431 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836568    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7e02a4de-6df2-4930-8360-4c5e09897472-sys-fs\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:38:57.837431 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836594    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-host-run-k8s-cni-cncf-io\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.837431 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836604    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-multus-socket-dir-parent\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.837431 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836483    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-multus-cni-dir\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.837431 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836622    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-host-run-netns\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.837431 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836619    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-multus-daemon-config\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.837431 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836638    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-sys\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.837431 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836648    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-host-run-k8s-cni-cncf-io\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.837431 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836596    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0a5d9c55-0799-4ae1-a902-dc06f813907c-ovnkube-script-lib\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.837431 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836650    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-host-run-netns\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.837431 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836680    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-host-var-lib-cni-multus\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.837431 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836706    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-host\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.837431 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836717    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-sysctl-d\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.838218 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836741    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-multus-conf-dir\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.838218 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836764    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-multus-conf-dir\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.838218 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836778    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-host-var-lib-cni-multus\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.838218 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836786    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-kubelet\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.838218 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836803    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-run-netns\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.838218 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836812    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-kubelet\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.838218 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836824    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-run-netns\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.838218 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836828    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b529371-cf9d-40e4-a2c5-55d69ab9a08d-host-slash\") pod \"iptables-alerter-689v8\" (UID: \"6b529371-cf9d-40e4-a2c5-55d69ab9a08d\") " pod="openshift-network-operator/iptables-alerter-689v8"
Nov 03 08:38:57.838218 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836860    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-host-var-lib-cni-bin\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.838218 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836863    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b529371-cf9d-40e4-a2c5-55d69ab9a08d-host-slash\") pod \"iptables-alerter-689v8\" (UID: \"6b529371-cf9d-40e4-a2c5-55d69ab9a08d\") " pod="openshift-network-operator/iptables-alerter-689v8"
Nov 03 08:38:57.838218 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836883    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcm5z\" (UniqueName: \"kubernetes.io/projected/f6ca3dc3-231e-4a8d-a16c-7384c225110a-kube-api-access-fcm5z\") pod \"network-check-target-clx84\" (UID: \"f6ca3dc3-231e-4a8d-a16c-7384c225110a\") " pod="openshift-network-diagnostics/network-check-target-clx84"
Nov 03 08:38:57.838218 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836900    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0a5d9c55-0799-4ae1-a902-dc06f813907c-ovnkube-config\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.838218 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836914    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-cni-binary-copy\") pod \"multus-additional-cni-plugins-wsrmr\" (UID: \"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9\") " pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:38:57.838218 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836886    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-host-var-lib-cni-bin\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.838218 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.836956    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7e02a4de-6df2-4930-8360-4c5e09897472-socket-dir\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:38:57.838218 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837001    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/906c75b6-325c-4e89-8a10-e4e355884133-dbus\") pod \"global-pull-secret-syncer-x9jpw\" (UID: \"906c75b6-325c-4e89-8a10-e4e355884133\") " pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:38:57.838218 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837024    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-var-lib-openvswitch\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.838765 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837045    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0a5d9c55-0799-4ae1-a902-dc06f813907c-ovn-node-metrics-cert\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.838765 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837056    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7e02a4de-6df2-4930-8360-4c5e09897472-socket-dir\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:38:57.838765 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837071    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-run-ovn-kubernetes\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.838765 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837095    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7e02a4de-6df2-4930-8360-4c5e09897472-registration-dir\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:38:57.838765 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837117    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6b529371-cf9d-40e4-a2c5-55d69ab9a08d-iptables-alerter-script\") pod \"iptables-alerter-689v8\" (UID: \"6b529371-cf9d-40e4-a2c5-55d69ab9a08d\") " pod="openshift-network-operator/iptables-alerter-689v8"
Nov 03 08:38:57.838765 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837135    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/110651f3-9a70-43bf-888d-0813682bc891-serviceca\") pod \"node-ca-5drgw\" (UID: \"110651f3-9a70-43bf-888d-0813682bc891\") " pod="openshift-image-registry/node-ca-5drgw"
Nov 03 08:38:57.838765 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837139    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-var-lib-openvswitch\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.838765 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837152    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-cnibin\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.838765 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837172    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/906c75b6-325c-4e89-8a10-e4e355884133-dbus\") pod \"global-pull-secret-syncer-x9jpw\" (UID: \"906c75b6-325c-4e89-8a10-e4e355884133\") " pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:38:57.838765 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837182    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-hostroot\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.838765 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837194    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-cnibin\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.838765 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837206    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-etc-kubernetes\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.838765 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837228    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7e02a4de-6df2-4930-8360-4c5e09897472-registration-dir\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:38:57.838765 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837231    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/906c75b6-325c-4e89-8a10-e4e355884133-kubelet-config\") pod \"global-pull-secret-syncer-x9jpw\" (UID: \"906c75b6-325c-4e89-8a10-e4e355884133\") " pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:38:57.838765 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837262    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/110651f3-9a70-43bf-888d-0813682bc891-host\") pod \"node-ca-5drgw\" (UID: \"110651f3-9a70-43bf-888d-0813682bc891\") " pod="openshift-image-registry/node-ca-5drgw"
Nov 03 08:38:57.838765 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837285    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-etc-openvswitch\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.838765 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837289    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/906c75b6-325c-4e89-8a10-e4e355884133-kubelet-config\") pod \"global-pull-secret-syncer-x9jpw\" (UID: \"906c75b6-325c-4e89-8a10-e4e355884133\") " pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:38:57.838765 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837303    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-run-ovn\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.839367 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837332    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-log-socket\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.839367 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837324    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-run-ovn\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.839367 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837355    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-modprobe-d\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.839367 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837376    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-systemd\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.839367 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837387    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-run-ovn-kubernetes\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.839367 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837397    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/801881d6-f217-4c80-a190-af953df3af04-tmp-dir\") pod \"node-resolver-hgbb5\" (UID: \"801881d6-f217-4c80-a190-af953df3af04\") " pod="openshift-dns/node-resolver-hgbb5"
Nov 03 08:38:57.839367 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837409    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/110651f3-9a70-43bf-888d-0813682bc891-host\") pod \"node-ca-5drgw\" (UID: \"110651f3-9a70-43bf-888d-0813682bc891\") " pod="openshift-image-registry/node-ca-5drgw"
Nov 03 08:38:57.839367 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837418    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bg9b\" (UniqueName: \"kubernetes.io/projected/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-kube-api-access-8bg9b\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.839367 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837423    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0a5d9c55-0799-4ae1-a902-dc06f813907c-ovnkube-config\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.839367 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837423    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-cni-binary-copy\") pod \"multus-additional-cni-plugins-wsrmr\" (UID: \"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9\") " pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:38:57.839367 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837439    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-systemd-units\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.839367 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837440    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-etc-openvswitch\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.839367 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837460    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-cni-netd\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.839367 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837469    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-hostroot\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.839367 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837473    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-etc-kubernetes\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.839367 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837505    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-systemd-units\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.839367 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837516    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-systemd\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.839367 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837509    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-cni-netd\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.839914 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837530    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-log-socket\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.839914 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837482    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-tuned\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.839914 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837564    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1649ad9-921c-49d9-baa5-c51f44175b8c-tmp\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.839914 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837586    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wsrmr\" (UID: \"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9\") " pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:38:57.839914 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837607    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wsrmr\" (UID: \"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9\") " pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:38:57.839914 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837625    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6b529371-cf9d-40e4-a2c5-55d69ab9a08d-iptables-alerter-script\") pod \"iptables-alerter-689v8\" (UID: \"6b529371-cf9d-40e4-a2c5-55d69ab9a08d\") " pod="openshift-network-operator/iptables-alerter-689v8"
Nov 03 08:38:57.839914 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837629    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0a5d9c55-0799-4ae1-a902-dc06f813907c-env-overrides\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.839914 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837713    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wsrmr\" (UID: \"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9\") " pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:38:57.839914 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837767    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/801881d6-f217-4c80-a190-af953df3af04-tmp-dir\") pod \"node-resolver-hgbb5\" (UID: \"801881d6-f217-4c80-a190-af953df3af04\") " pod="openshift-dns/node-resolver-hgbb5"
Nov 03 08:38:57.839914 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837772    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-system-cni-dir\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.839914 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837798    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/110651f3-9a70-43bf-888d-0813682bc891-serviceca\") pod \"node-ca-5drgw\" (UID: \"110651f3-9a70-43bf-888d-0813682bc891\") " pod="openshift-image-registry/node-ca-5drgw"
Nov 03 08:38:57.839914 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837767    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wsrmr\" (UID: \"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9\") " pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:38:57.839914 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837868    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-modprobe-d\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.839914 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837910    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-system-cni-dir\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.839914 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837798    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e1386760-3800-418f-bb1b-20e19880620b-konnectivity-ca\") pod \"konnectivity-agent-bq5p4\" (UID: \"e1386760-3800-418f-bb1b-20e19880620b\") " pod="kube-system/konnectivity-agent-bq5p4"
Nov 03 08:38:57.839914 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837967    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-host-var-lib-kubelet\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.839914 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.837992    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnnxt\" (UniqueName: \"kubernetes.io/projected/d1649ad9-921c-49d9-baa5-c51f44175b8c-kube-api-access-xnnxt\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.840431 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.838031    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-cnibin\") pod \"multus-additional-cni-plugins-wsrmr\" (UID: \"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9\") " pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:38:57.840431 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.838094    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-cnibin\") pod \"multus-additional-cni-plugins-wsrmr\" (UID: \"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9\") " pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:38:57.840431 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.838122    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-host-var-lib-kubelet\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.840431 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.838928    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wsrmr\" (UID: \"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9\") " pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:38:57.840431 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.839133    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0a5d9c55-0799-4ae1-a902-dc06f813907c-env-overrides\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.840431 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.839768    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wsrmr\" (UID: \"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9\") " pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:38:57.840431 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.839843    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e1386760-3800-418f-bb1b-20e19880620b-konnectivity-ca\") pod \"konnectivity-agent-bq5p4\" (UID: \"e1386760-3800-418f-bb1b-20e19880620b\") " pod="kube-system/konnectivity-agent-bq5p4"
Nov 03 08:38:57.840431 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.840242    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1649ad9-921c-49d9-baa5-c51f44175b8c-tmp\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.840431 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.840296    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-tuned\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.840431 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.840322    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0a5d9c55-0799-4ae1-a902-dc06f813907c-ovn-node-metrics-cert\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.840678 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.840437    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e1386760-3800-418f-bb1b-20e19880620b-agent-certs\") pod \"konnectivity-agent-bq5p4\" (UID: \"e1386760-3800-418f-bb1b-20e19880620b\") " pod="kube-system/konnectivity-agent-bq5p4"
Nov 03 08:38:57.890082 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.889727    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvsbx\" (UniqueName: \"kubernetes.io/projected/7e02a4de-6df2-4930-8360-4c5e09897472-kube-api-access-jvsbx\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:38:57.890082 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.889782    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99jxr\" (UniqueName: \"kubernetes.io/projected/7c26dfa4-e47e-43a1-94a2-816d22fb7f3a-kube-api-access-99jxr\") pod \"network-metrics-daemon-pflts\" (UID: \"7c26dfa4-e47e-43a1-94a2-816d22fb7f3a\") " pod="openshift-multus/network-metrics-daemon-pflts"
Nov 03 08:38:57.890082 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:57.889960    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 08:38:57.890082 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:57.889979    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 08:38:57.890082 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:57.889991    2575 projected.go:196] Error preparing data for projected volume kube-api-access-fcm5z for pod openshift-network-diagnostics/network-check-target-clx84: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 08:38:57.890082 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:57.890041    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6ca3dc3-231e-4a8d-a16c-7384c225110a-kube-api-access-fcm5z podName:f6ca3dc3-231e-4a8d-a16c-7384c225110a nodeName:}" failed. No retries permitted until 2026-11-03 08:38:58.390026058 +0000 UTC m=+3.163646103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fcm5z" (UniqueName: "kubernetes.io/projected/f6ca3dc3-231e-4a8d-a16c-7384c225110a-kube-api-access-fcm5z") pod "network-check-target-clx84" (UID: "f6ca3dc3-231e-4a8d-a16c-7384c225110a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 08:38:57.890595 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.890565    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4vs4\" (UniqueName: \"kubernetes.io/projected/0a5d9c55-0799-4ae1-a902-dc06f813907c-kube-api-access-s4vs4\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:57.891392 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.891365    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njvj8\" (UniqueName: \"kubernetes.io/projected/801881d6-f217-4c80-a190-af953df3af04-kube-api-access-njvj8\") pod \"node-resolver-hgbb5\" (UID: \"801881d6-f217-4c80-a190-af953df3af04\") " pod="openshift-dns/node-resolver-hgbb5"
Nov 03 08:38:57.891514 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.891500    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnnxt\" (UniqueName: \"kubernetes.io/projected/d1649ad9-921c-49d9-baa5-c51f44175b8c-kube-api-access-xnnxt\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:57.891782 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.891766    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bg9b\" (UniqueName: \"kubernetes.io/projected/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-kube-api-access-8bg9b\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:57.892084 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.892061    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpfx2\" (UniqueName: \"kubernetes.io/projected/110651f3-9a70-43bf-888d-0813682bc891-kube-api-access-hpfx2\") pod \"node-ca-5drgw\" (UID: \"110651f3-9a70-43bf-888d-0813682bc891\") " pod="openshift-image-registry/node-ca-5drgw"
Nov 03 08:38:57.892265 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.892252    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgfkn\" (UniqueName: \"kubernetes.io/projected/6b529371-cf9d-40e4-a2c5-55d69ab9a08d-kube-api-access-sgfkn\") pod \"iptables-alerter-689v8\" (UID: \"6b529371-cf9d-40e4-a2c5-55d69ab9a08d\") " pod="openshift-network-operator/iptables-alerter-689v8"
Nov 03 08:38:57.892299 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.892254    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rxtf\" (UniqueName: \"kubernetes.io/projected/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-kube-api-access-7rxtf\") pod \"multus-additional-cni-plugins-wsrmr\" (UID: \"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9\") " pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:38:57.893678 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.893662    2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 08:38:57.893729 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:57.893712    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ldw6t"
Nov 03 08:38:58.006244 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:58.006220    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:38:58.007284 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:58.007266    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5drgw"
Nov 03 08:38:58.014066 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:58.014044    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e02a4de_6df2_4930_8360_4c5e09897472.slice/crio-977e7973a68f0a7bba883c7e4ec44a6f55a546c7f6a168b507f771f85a13c3ab WatchSource:0}: Error finding container 977e7973a68f0a7bba883c7e4ec44a6f55a546c7f6a168b507f771f85a13c3ab: Status 404 returned error can't find the container with id 977e7973a68f0a7bba883c7e4ec44a6f55a546c7f6a168b507f771f85a13c3ab
Nov 03 08:38:58.015274 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:58.015259    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kzcqg"
Nov 03 08:38:58.015617 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:58.015602    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod110651f3_9a70_43bf_888d_0813682bc891.slice/crio-da7a63703f623ef740358ec23a7ea0a5619ff7c821233a8c0790d67dbfb5661f WatchSource:0}: Error finding container da7a63703f623ef740358ec23a7ea0a5619ff7c821233a8c0790d67dbfb5661f: Status 404 returned error can't find the container with id da7a63703f623ef740358ec23a7ea0a5619ff7c821233a8c0790d67dbfb5661f
Nov 03 08:38:58.019525 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:58.019515    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-689v8"
Nov 03 08:38:58.023132 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:58.023112    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod105e26de_f1f1_4aae_99ef_e6a9491ccbe8.slice/crio-3ddf1aaa0fa51521a2ebb29aada9c9536366809baf767c1d14faec6b34055f75 WatchSource:0}: Error finding container 3ddf1aaa0fa51521a2ebb29aada9c9536366809baf767c1d14faec6b34055f75: Status 404 returned error can't find the container with id 3ddf1aaa0fa51521a2ebb29aada9c9536366809baf767c1d14faec6b34055f75
Nov 03 08:38:58.027078 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:58.027041    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b529371_cf9d_40e4_a2c5_55d69ab9a08d.slice/crio-de355649162da193e299f0a349b7f80afd241a4b5c7f8dfb7b38d5dfb7077678 WatchSource:0}: Error finding container de355649162da193e299f0a349b7f80afd241a4b5c7f8dfb7b38d5dfb7077678: Status 404 returned error can't find the container with id de355649162da193e299f0a349b7f80afd241a4b5c7f8dfb7b38d5dfb7077678
Nov 03 08:38:58.036789 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:58.036774    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hgbb5"
Nov 03 08:38:58.039197 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:58.039175    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bq5p4"
Nov 03 08:38:58.044595 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:58.044564    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:38:58.045979 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:58.045921    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:38:58.046056 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:58.046012    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod801881d6_f217_4c80_a190_af953df3af04.slice/crio-e6ed42962fc5432b2d8e7c837373b8999404b2c6b6361d2f782824a103e27830 WatchSource:0}: Error finding container e6ed42962fc5432b2d8e7c837373b8999404b2c6b6361d2f782824a103e27830: Status 404 returned error can't find the container with id e6ed42962fc5432b2d8e7c837373b8999404b2c6b6361d2f782824a103e27830
Nov 03 08:38:58.054355 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:58.054333    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1386760_3800_418f_bb1b_20e19880620b.slice/crio-d3f5442cc25a0226fe87ab23a8535a38b81051a06246180dca6beacd8ab7d17e WatchSource:0}: Error finding container d3f5442cc25a0226fe87ab23a8535a38b81051a06246180dca6beacd8ab7d17e: Status 404 returned error can't find the container with id d3f5442cc25a0226fe87ab23a8535a38b81051a06246180dca6beacd8ab7d17e
Nov 03 08:38:58.060541 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:58.060277    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a5d9c55_0799_4ae1_a902_dc06f813907c.slice/crio-e7f26d2e538a5750f84d313224064ca989b465ae6d4252c818e516fdbdfa8310 WatchSource:0}: Error finding container e7f26d2e538a5750f84d313224064ca989b465ae6d4252c818e516fdbdfa8310: Status 404 returned error can't find the container with id e7f26d2e538a5750f84d313224064ca989b465ae6d4252c818e516fdbdfa8310
Nov 03 08:38:58.060541 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:58.060418    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:38:58.070319 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:38:58.070300    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968e5615_5e12_4dbf_b14d_e1fdb1e1cbe9.slice/crio-c020da724b255d14c6a1e90b1d4cfa4e3f76de617fcd0ceaca2564f8abcc2c22 WatchSource:0}: Error finding container c020da724b255d14c6a1e90b1d4cfa4e3f76de617fcd0ceaca2564f8abcc2c22: Status 404 returned error can't find the container with id c020da724b255d14c6a1e90b1d4cfa4e3f76de617fcd0ceaca2564f8abcc2c22
Nov 03 08:38:58.342579 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:58.342497    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/906c75b6-325c-4e89-8a10-e4e355884133-original-pull-secret\") pod \"global-pull-secret-syncer-x9jpw\" (UID: \"906c75b6-325c-4e89-8a10-e4e355884133\") " pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:38:58.342579 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:58.342553    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c26dfa4-e47e-43a1-94a2-816d22fb7f3a-metrics-certs\") pod \"network-metrics-daemon-pflts\" (UID: \"7c26dfa4-e47e-43a1-94a2-816d22fb7f3a\") " pod="openshift-multus/network-metrics-daemon-pflts"
Nov 03 08:38:58.342844 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:58.342646    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 08:38:58.342844 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:58.342708    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/906c75b6-325c-4e89-8a10-e4e355884133-original-pull-secret podName:906c75b6-325c-4e89-8a10-e4e355884133 nodeName:}" failed. No retries permitted until 2026-11-03 08:38:59.342689789 +0000 UTC m=+4.116309820 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/906c75b6-325c-4e89-8a10-e4e355884133-original-pull-secret") pod "global-pull-secret-syncer-x9jpw" (UID: "906c75b6-325c-4e89-8a10-e4e355884133") : object "kube-system"/"original-pull-secret" not registered
Nov 03 08:38:58.342844 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:58.342649    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 08:38:58.342844 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:58.342745    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c26dfa4-e47e-43a1-94a2-816d22fb7f3a-metrics-certs podName:7c26dfa4-e47e-43a1-94a2-816d22fb7f3a nodeName:}" failed. No retries permitted until 2026-11-03 08:38:59.342739112 +0000 UTC m=+4.116359140 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c26dfa4-e47e-43a1-94a2-816d22fb7f3a-metrics-certs") pod "network-metrics-daemon-pflts" (UID: "7c26dfa4-e47e-43a1-94a2-816d22fb7f3a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 08:38:58.442880 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:58.442847    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcm5z\" (UniqueName: \"kubernetes.io/projected/f6ca3dc3-231e-4a8d-a16c-7384c225110a-kube-api-access-fcm5z\") pod \"network-check-target-clx84\" (UID: \"f6ca3dc3-231e-4a8d-a16c-7384c225110a\") " pod="openshift-network-diagnostics/network-check-target-clx84"
Nov 03 08:38:58.443044 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:58.443027    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 08:38:58.443044 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:58.443043    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 08:38:58.443157 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:58.443086    2575 projected.go:196] Error preparing data for projected volume kube-api-access-fcm5z for pod openshift-network-diagnostics/network-check-target-clx84: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 08:38:58.443204 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:58.443179    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6ca3dc3-231e-4a8d-a16c-7384c225110a-kube-api-access-fcm5z podName:f6ca3dc3-231e-4a8d-a16c-7384c225110a nodeName:}" failed. No retries permitted until 2026-11-03 08:38:59.443159294 +0000 UTC m=+4.216779327 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fcm5z" (UniqueName: "kubernetes.io/projected/f6ca3dc3-231e-4a8d-a16c-7384c225110a-kube-api-access-fcm5z") pod "network-check-target-clx84" (UID: "f6ca3dc3-231e-4a8d-a16c-7384c225110a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 08:38:58.711592 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:58.711508    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wsrmr" event={"ID":"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9","Type":"ContainerStarted","Data":"c020da724b255d14c6a1e90b1d4cfa4e3f76de617fcd0ceaca2564f8abcc2c22"}
Nov 03 08:38:58.712979 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:58.712934    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt" event={"ID":"0a5d9c55-0799-4ae1-a902-dc06f813907c","Type":"ContainerStarted","Data":"e7f26d2e538a5750f84d313224064ca989b465ae6d4252c818e516fdbdfa8310"}
Nov 03 08:38:58.715106 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:58.715081    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jpll8" event={"ID":"d1649ad9-921c-49d9-baa5-c51f44175b8c","Type":"ContainerStarted","Data":"02cbc745ea0a1b9f603aaeafcefd2a10220e3bae9c814263543722c28080909b"}
Nov 03 08:38:58.716484 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:58.716461    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hgbb5" event={"ID":"801881d6-f217-4c80-a190-af953df3af04","Type":"ContainerStarted","Data":"e6ed42962fc5432b2d8e7c837373b8999404b2c6b6361d2f782824a103e27830"}
Nov 03 08:38:58.717972 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:58.717935    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5drgw" event={"ID":"110651f3-9a70-43bf-888d-0813682bc891","Type":"ContainerStarted","Data":"da7a63703f623ef740358ec23a7ea0a5619ff7c821233a8c0790d67dbfb5661f"}
Nov 03 08:38:58.719041 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:58.719001    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bq5p4" event={"ID":"e1386760-3800-418f-bb1b-20e19880620b","Type":"ContainerStarted","Data":"d3f5442cc25a0226fe87ab23a8535a38b81051a06246180dca6beacd8ab7d17e"}
Nov 03 08:38:58.720053 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:58.720027    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-689v8" event={"ID":"6b529371-cf9d-40e4-a2c5-55d69ab9a08d","Type":"ContainerStarted","Data":"de355649162da193e299f0a349b7f80afd241a4b5c7f8dfb7b38d5dfb7077678"}
Nov 03 08:38:58.721271 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:58.721252    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kzcqg" event={"ID":"105e26de-f1f1-4aae-99ef-e6a9491ccbe8","Type":"ContainerStarted","Data":"3ddf1aaa0fa51521a2ebb29aada9c9536366809baf767c1d14faec6b34055f75"}
Nov 03 08:38:58.722653 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:58.722622    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs" event={"ID":"7e02a4de-6df2-4930-8360-4c5e09897472","Type":"ContainerStarted","Data":"977e7973a68f0a7bba883c7e4ec44a6f55a546c7f6a168b507f771f85a13c3ab"}
Nov 03 08:38:58.898002 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:58.897959    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 08:33:57 +0000 UTC" deadline="2028-06-20 21:01:00.981787764 +0000 UTC"
Nov 03 08:38:58.898002 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:58.897993    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14292h22m2.083798634s"
Nov 03 08:38:59.352886 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:59.352850    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c26dfa4-e47e-43a1-94a2-816d22fb7f3a-metrics-certs\") pod \"network-metrics-daemon-pflts\" (UID: \"7c26dfa4-e47e-43a1-94a2-816d22fb7f3a\") " pod="openshift-multus/network-metrics-daemon-pflts"
Nov 03 08:38:59.353043 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:59.352963    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/906c75b6-325c-4e89-8a10-e4e355884133-original-pull-secret\") pod \"global-pull-secret-syncer-x9jpw\" (UID: \"906c75b6-325c-4e89-8a10-e4e355884133\") " pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:38:59.353125 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:59.353106    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 08:38:59.353209 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:59.353175    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/906c75b6-325c-4e89-8a10-e4e355884133-original-pull-secret podName:906c75b6-325c-4e89-8a10-e4e355884133 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:01.353157937 +0000 UTC m=+6.126777973 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/906c75b6-325c-4e89-8a10-e4e355884133-original-pull-secret") pod "global-pull-secret-syncer-x9jpw" (UID: "906c75b6-325c-4e89-8a10-e4e355884133") : object "kube-system"/"original-pull-secret" not registered
Nov 03 08:38:59.353460 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:59.353277    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 08:38:59.353460 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:59.353338    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c26dfa4-e47e-43a1-94a2-816d22fb7f3a-metrics-certs podName:7c26dfa4-e47e-43a1-94a2-816d22fb7f3a nodeName:}" failed. No retries permitted until 2026-11-03 08:39:01.353303124 +0000 UTC m=+6.126923158 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c26dfa4-e47e-43a1-94a2-816d22fb7f3a-metrics-certs") pod "network-metrics-daemon-pflts" (UID: "7c26dfa4-e47e-43a1-94a2-816d22fb7f3a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 08:38:59.453403 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:59.453369    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcm5z\" (UniqueName: \"kubernetes.io/projected/f6ca3dc3-231e-4a8d-a16c-7384c225110a-kube-api-access-fcm5z\") pod \"network-check-target-clx84\" (UID: \"f6ca3dc3-231e-4a8d-a16c-7384c225110a\") " pod="openshift-network-diagnostics/network-check-target-clx84"
Nov 03 08:38:59.453691 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:59.453550    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 08:38:59.453691 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:59.453569    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 08:38:59.453691 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:59.453580    2575 projected.go:196] Error preparing data for projected volume kube-api-access-fcm5z for pod openshift-network-diagnostics/network-check-target-clx84: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 08:38:59.453691 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:59.453634    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6ca3dc3-231e-4a8d-a16c-7384c225110a-kube-api-access-fcm5z podName:f6ca3dc3-231e-4a8d-a16c-7384c225110a nodeName:}" failed. No retries permitted until 2026-11-03 08:39:01.453617001 +0000 UTC m=+6.227237037 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fcm5z" (UniqueName: "kubernetes.io/projected/f6ca3dc3-231e-4a8d-a16c-7384c225110a-kube-api-access-fcm5z") pod "network-check-target-clx84" (UID: "f6ca3dc3-231e-4a8d-a16c-7384c225110a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 08:38:59.706496 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:59.704542    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-clx84"
Nov 03 08:38:59.706496 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:59.704676    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-clx84" podUID="f6ca3dc3-231e-4a8d-a16c-7384c225110a"
Nov 03 08:38:59.706496 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:59.704763    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pflts"
Nov 03 08:38:59.706496 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:59.704850    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pflts" podUID="7c26dfa4-e47e-43a1-94a2-816d22fb7f3a"
Nov 03 08:38:59.706496 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:59.706216    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:38:59.706496 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:38:59.706315    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9jpw" podUID="906c75b6-325c-4e89-8a10-e4e355884133"
Nov 03 08:38:59.898336 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:59.898253    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 08:33:57 +0000 UTC" deadline="2028-04-21 22:59:37.646813371 +0000 UTC"
Nov 03 08:38:59.898336 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:38:59.898314    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12854h20m37.748504021s"
Nov 03 08:39:00.738999 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:00.738627    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-73.us-west-1.compute.internal" event={"ID":"0f8947715126d508f609643819d8b2a7","Type":"ContainerStarted","Data":"5a3a3ea589cb9f3b3144012d6d55ea14cb7924dd6ebaae7b63351e1c57225b58"}
Nov 03 08:39:00.751129 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:00.751034    2575 generic.go:356] "Generic (PLEG): container finished" podID="bda1773f2a5825c17c394ccc2e58c1ce" containerID="89c2c10538271e6aa71010aab9533709f01fc8653ca6ff409f44c310fa997afb" exitCode=0
Nov 03 08:39:00.751129 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:00.751084    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal" event={"ID":"bda1773f2a5825c17c394ccc2e58c1ce","Type":"ContainerDied","Data":"89c2c10538271e6aa71010aab9533709f01fc8653ca6ff409f44c310fa997afb"}
Nov 03 08:39:00.969005 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:00.968934    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-73.us-west-1.compute.internal" podStartSLOduration=3.968915878 podStartE2EDuration="3.968915878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 08:38:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:00.853651462 +0000 UTC m=+5.627271516" watchObservedRunningTime="2026-11-03 08:39:00.968915878 +0000 UTC m=+5.742535932"
Nov 03 08:39:01.369456 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:01.369034    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c26dfa4-e47e-43a1-94a2-816d22fb7f3a-metrics-certs\") pod \"network-metrics-daemon-pflts\" (UID: \"7c26dfa4-e47e-43a1-94a2-816d22fb7f3a\") " pod="openshift-multus/network-metrics-daemon-pflts"
Nov 03 08:39:01.369456 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:01.369183    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/906c75b6-325c-4e89-8a10-e4e355884133-original-pull-secret\") pod \"global-pull-secret-syncer-x9jpw\" (UID: \"906c75b6-325c-4e89-8a10-e4e355884133\") " pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:39:01.369456 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:01.369221    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 08:39:01.369456 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:01.369262    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 08:39:01.369456 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:01.369278    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c26dfa4-e47e-43a1-94a2-816d22fb7f3a-metrics-certs podName:7c26dfa4-e47e-43a1-94a2-816d22fb7f3a nodeName:}" failed. No retries permitted until 2026-11-03 08:39:05.36925939 +0000 UTC m=+10.142879444 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c26dfa4-e47e-43a1-94a2-816d22fb7f3a-metrics-certs") pod "network-metrics-daemon-pflts" (UID: "7c26dfa4-e47e-43a1-94a2-816d22fb7f3a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 08:39:01.369456 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:01.369295    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/906c75b6-325c-4e89-8a10-e4e355884133-original-pull-secret podName:906c75b6-325c-4e89-8a10-e4e355884133 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:05.369284546 +0000 UTC m=+10.142904591 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/906c75b6-325c-4e89-8a10-e4e355884133-original-pull-secret") pod "global-pull-secret-syncer-x9jpw" (UID: "906c75b6-325c-4e89-8a10-e4e355884133") : object "kube-system"/"original-pull-secret" not registered
Nov 03 08:39:01.470159 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:01.470125    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcm5z\" (UniqueName: \"kubernetes.io/projected/f6ca3dc3-231e-4a8d-a16c-7384c225110a-kube-api-access-fcm5z\") pod \"network-check-target-clx84\" (UID: \"f6ca3dc3-231e-4a8d-a16c-7384c225110a\") " pod="openshift-network-diagnostics/network-check-target-clx84"
Nov 03 08:39:01.470323 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:01.470313    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 08:39:01.470400 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:01.470331    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 08:39:01.470400 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:01.470343    2575 projected.go:196] Error preparing data for projected volume kube-api-access-fcm5z for pod openshift-network-diagnostics/network-check-target-clx84: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 08:39:01.470400 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:01.470394    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6ca3dc3-231e-4a8d-a16c-7384c225110a-kube-api-access-fcm5z podName:f6ca3dc3-231e-4a8d-a16c-7384c225110a nodeName:}" failed. No retries permitted until 2026-11-03 08:39:05.470377655 +0000 UTC m=+10.243997691 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fcm5z" (UniqueName: "kubernetes.io/projected/f6ca3dc3-231e-4a8d-a16c-7384c225110a-kube-api-access-fcm5z") pod "network-check-target-clx84" (UID: "f6ca3dc3-231e-4a8d-a16c-7384c225110a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 08:39:01.705431 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:01.705362    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-clx84"
Nov 03 08:39:01.705586 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:01.705486    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-clx84" podUID="f6ca3dc3-231e-4a8d-a16c-7384c225110a"
Nov 03 08:39:01.705850 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:01.705831    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:39:01.705965 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:01.705920    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9jpw" podUID="906c75b6-325c-4e89-8a10-e4e355884133"
Nov 03 08:39:01.706131 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:01.706092    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pflts"
Nov 03 08:39:01.706247 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:01.706191    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pflts" podUID="7c26dfa4-e47e-43a1-94a2-816d22fb7f3a"
Nov 03 08:39:01.768864 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:01.768152    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal" event={"ID":"bda1773f2a5825c17c394ccc2e58c1ce","Type":"ContainerStarted","Data":"99cb3009262cc0a9abb3e55da11a02674c59bb9911766c21ed86a962bc5927b2"}
Nov 03 08:39:01.773221 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:01.772557    2575 generic.go:356] "Generic (PLEG): container finished" podID="968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9" containerID="1542ec1a8594d66dea43650dc095dad2d387f3ffe951d7a566916aba30b855db" exitCode=0
Nov 03 08:39:01.773221 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:01.772708    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wsrmr" event={"ID":"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9","Type":"ContainerDied","Data":"1542ec1a8594d66dea43650dc095dad2d387f3ffe951d7a566916aba30b855db"}
Nov 03 08:39:02.004296 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:02.004196    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal" podStartSLOduration=5.004176905 podStartE2EDuration="5.004176905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 08:38:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:01.883469345 +0000 UTC m=+6.657089399" watchObservedRunningTime="2026-11-03 08:39:02.004176905 +0000 UTC m=+6.777796960"
Nov 03 08:39:02.780131 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:02.779993    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bq5p4" event={"ID":"e1386760-3800-418f-bb1b-20e19880620b","Type":"ContainerStarted","Data":"74005954619fa2a0ecff3c319c3e36108ae69f3a268ceb7a19b2aeb4192d1c90"}
Nov 03 08:39:03.704191 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:03.704125    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pflts"
Nov 03 08:39:03.704617 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:03.704202    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:39:03.704617 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:03.704222    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-clx84"
Nov 03 08:39:03.704617 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:03.704340    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9jpw" podUID="906c75b6-325c-4e89-8a10-e4e355884133"
Nov 03 08:39:03.704907 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:03.704877    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-clx84" podUID="f6ca3dc3-231e-4a8d-a16c-7384c225110a"
Nov 03 08:39:03.705077 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:03.705050    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pflts" podUID="7c26dfa4-e47e-43a1-94a2-816d22fb7f3a"
Nov 03 08:39:03.787446 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:03.787415    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5drgw" event={"ID":"110651f3-9a70-43bf-888d-0813682bc891","Type":"ContainerStarted","Data":"a4e4a7eb481d75aadc727c2425ec7f7e6816b2dee26a6272cef035857b92a93f"}
Nov 03 08:39:03.907865 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:03.907801    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bq5p4" podStartSLOduration=3.481620936 podStartE2EDuration="7.907779446s" totalImagesPullingTime="4.42615851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:02.894727573 +0000 UTC m=+7.668347627" watchObservedRunningTime="2026-11-03 08:39:03.907779446 +0000 UTC m=+8.681399504"
Nov 03 08:39:04.791275 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:04.791237    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs" event={"ID":"7e02a4de-6df2-4930-8360-4c5e09897472","Type":"ContainerStarted","Data":"fd2618169d0271be4acc6080ba92baea5f73433984514d2f34b745136cd73456"}
Nov 03 08:39:05.239512 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:05.239010    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bq5p4"
Nov 03 08:39:05.239643 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:05.239546    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bq5p4"
Nov 03 08:39:05.354657 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:05.354596    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5drgw" podStartSLOduration=3.964992944 podStartE2EDuration="9.354578623s" totalImagesPullingTime="5.389585679s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 08:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:03.908060682 +0000 UTC m=+8.681680735" watchObservedRunningTime="2026-11-03 08:39:05.354578623 +0000 UTC m=+10.128198678"
Nov 03 08:39:05.402155 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:05.401965    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/906c75b6-325c-4e89-8a10-e4e355884133-original-pull-secret\") pod \"global-pull-secret-syncer-x9jpw\" (UID: \"906c75b6-325c-4e89-8a10-e4e355884133\") " pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:39:05.402155 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:05.402020    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c26dfa4-e47e-43a1-94a2-816d22fb7f3a-metrics-certs\") pod \"network-metrics-daemon-pflts\" (UID: \"7c26dfa4-e47e-43a1-94a2-816d22fb7f3a\") " pod="openshift-multus/network-metrics-daemon-pflts"
Nov 03 08:39:05.402155 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:05.402073    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 08:39:05.402155 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:05.402110    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 08:39:05.402155 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:05.402148    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/906c75b6-325c-4e89-8a10-e4e355884133-original-pull-secret podName:906c75b6-325c-4e89-8a10-e4e355884133 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:13.402132013 +0000 UTC m=+18.175752058 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/906c75b6-325c-4e89-8a10-e4e355884133-original-pull-secret") pod "global-pull-secret-syncer-x9jpw" (UID: "906c75b6-325c-4e89-8a10-e4e355884133") : object "kube-system"/"original-pull-secret" not registered
Nov 03 08:39:05.402155 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:05.402166    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c26dfa4-e47e-43a1-94a2-816d22fb7f3a-metrics-certs podName:7c26dfa4-e47e-43a1-94a2-816d22fb7f3a nodeName:}" failed. No retries permitted until 2026-11-03 08:39:13.402158701 +0000 UTC m=+18.175778733 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c26dfa4-e47e-43a1-94a2-816d22fb7f3a-metrics-certs") pod "network-metrics-daemon-pflts" (UID: "7c26dfa4-e47e-43a1-94a2-816d22fb7f3a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 08:39:05.502885 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:05.502765    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcm5z\" (UniqueName: \"kubernetes.io/projected/f6ca3dc3-231e-4a8d-a16c-7384c225110a-kube-api-access-fcm5z\") pod \"network-check-target-clx84\" (UID: \"f6ca3dc3-231e-4a8d-a16c-7384c225110a\") " pod="openshift-network-diagnostics/network-check-target-clx84"
Nov 03 08:39:05.503105 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:05.502934    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 08:39:05.503105 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:05.502974    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 08:39:05.503105 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:05.502988    2575 projected.go:196] Error preparing data for projected volume kube-api-access-fcm5z for pod openshift-network-diagnostics/network-check-target-clx84: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 08:39:05.503105 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:05.503042    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6ca3dc3-231e-4a8d-a16c-7384c225110a-kube-api-access-fcm5z podName:f6ca3dc3-231e-4a8d-a16c-7384c225110a nodeName:}" failed. No retries permitted until 2026-11-03 08:39:13.503026197 +0000 UTC m=+18.276646241 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fcm5z" (UniqueName: "kubernetes.io/projected/f6ca3dc3-231e-4a8d-a16c-7384c225110a-kube-api-access-fcm5z") pod "network-check-target-clx84" (UID: "f6ca3dc3-231e-4a8d-a16c-7384c225110a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 08:39:05.705891 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:05.705865    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pflts"
Nov 03 08:39:05.706325 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:05.706090    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:39:05.706325 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:05.706169    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-clx84"
Nov 03 08:39:05.706889 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:05.706781    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9jpw" podUID="906c75b6-325c-4e89-8a10-e4e355884133"
Nov 03 08:39:05.706889 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:05.706857    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-clx84" podUID="f6ca3dc3-231e-4a8d-a16c-7384c225110a"
Nov 03 08:39:05.707123 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:05.706700    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pflts" podUID="7c26dfa4-e47e-43a1-94a2-816d22fb7f3a"
Nov 03 08:39:06.363420 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:06.363396    2575 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 08:39:06.553346 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:06.553282    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bq5p4"
Nov 03 08:39:06.554030 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:06.554003    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bq5p4"
Nov 03 08:39:06.693134 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:06.693015    2575 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T08:39:06.363418518Z","UUID":"5946a8ad-de1c-4d28-b647-a88b89170621","Handler":null,"Name":"","Endpoint":""}
Nov 03 08:39:06.694493 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:06.694473    2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 08:39:06.694493 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:06.694500    2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 08:39:06.796900 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:06.796860    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kzcqg" event={"ID":"105e26de-f1f1-4aae-99ef-e6a9491ccbe8","Type":"ContainerStarted","Data":"eb2a174f321f16a2d455e0dd6aa7a87a8ee30a460754e0b361d0e15b1bad7507"}
Nov 03 08:39:06.799447 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:06.799418    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs" event={"ID":"7e02a4de-6df2-4930-8360-4c5e09897472","Type":"ContainerStarted","Data":"86e15fe9786b5b98599e09c89ce4f4871afd174bb7b010df9849738a7cb40d05"}
Nov 03 08:39:06.801680 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:06.801649    2575 generic.go:356] "Generic (PLEG): container finished" podID="968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9" containerID="090e8940649456277cd57142c0c6042a804ea921e5f62d3a371c971fbacd0c6f" exitCode=0
Nov 03 08:39:06.801763 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:06.801721    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wsrmr" event={"ID":"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9","Type":"ContainerDied","Data":"090e8940649456277cd57142c0c6042a804ea921e5f62d3a371c971fbacd0c6f"}
Nov 03 08:39:06.803220 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:06.803197    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hgbb5" event={"ID":"801881d6-f217-4c80-a190-af953df3af04","Type":"ContainerStarted","Data":"d4c8045e80de4a599c7b89f4904fb6c1f07529755a42151f421afc343bbb217d"}
Nov 03 08:39:07.043484 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:07.043390    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-kzcqg" podStartSLOduration=2.896769581 podStartE2EDuration="11.043368591s" totalImagesPullingTime="8.14659901s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 08:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:06.925126998 +0000 UTC m=+11.698747052" watchObservedRunningTime="2026-11-03 08:39:07.043368591 +0000 UTC m=+11.816988656"
Nov 03 08:39:07.162446 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:07.162392    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-hgbb5" podStartSLOduration=2.760622569 podStartE2EDuration="11.162376688s" totalImagesPullingTime="8.401754119s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 08:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:07.161744841 +0000 UTC m=+11.935364895" watchObservedRunningTime="2026-11-03 08:39:07.162376688 +0000 UTC m=+11.935996741"
Nov 03 08:39:07.703165 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:07.703122    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:39:07.703165 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:07.703149    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pflts"
Nov 03 08:39:07.703510 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:07.703224    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-clx84"
Nov 03 08:39:07.703510 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:07.703339    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9jpw" podUID="906c75b6-325c-4e89-8a10-e4e355884133"
Nov 03 08:39:07.703510 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:07.703424    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-clx84" podUID="f6ca3dc3-231e-4a8d-a16c-7384c225110a"
Nov 03 08:39:07.703510 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:07.703499    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pflts" podUID="7c26dfa4-e47e-43a1-94a2-816d22fb7f3a"
Nov 03 08:39:08.821448 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:08.821195    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-689v8" event={"ID":"6b529371-cf9d-40e4-a2c5-55d69ab9a08d","Type":"ContainerStarted","Data":"6683e93b9cf16bae4c3e6de59db36fa56d3cb58e0eee0988149cf85e07703920"}
Nov 03 08:39:08.823681 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:08.823652    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs" event={"ID":"7e02a4de-6df2-4930-8360-4c5e09897472","Type":"ContainerStarted","Data":"2c006c415cdced7c30c25a292accb589652e854954627422dc6a412cadc0c94c"}
Nov 03 08:39:08.825961 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:08.825922    2575 generic.go:356] "Generic (PLEG): container finished" podID="968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9" containerID="d905d4d84b96574c3372a719706b6cac6f68e70410fe1f9828519895412776ee" exitCode=0
Nov 03 08:39:08.826079 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:08.826018    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wsrmr" event={"ID":"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9","Type":"ContainerDied","Data":"d905d4d84b96574c3372a719706b6cac6f68e70410fe1f9828519895412776ee"}
Nov 03 08:39:08.827503 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:08.827477    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jpll8" event={"ID":"d1649ad9-921c-49d9-baa5-c51f44175b8c","Type":"ContainerStarted","Data":"b66139b098398d08611ebb1df549dfc0cd12d6f114804d57600c92ebf52b0fdf"}
Nov 03 08:39:08.937288 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:08.937235    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-689v8" podStartSLOduration=4.442390437 podStartE2EDuration="12.937218413s" totalImagesPullingTime="8.494827976s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 08:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:08.936639007 +0000 UTC m=+13.710259052" watchObservedRunningTime="2026-11-03 08:39:08.937218413 +0000 UTC m=+13.710838470"
Nov 03 08:39:09.052496 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:09.052441    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jpll8" podStartSLOduration=2.837220371 podStartE2EDuration="13.052429015s" totalImagesPullingTime="10.215208644s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 08:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:09.052111841 +0000 UTC m=+13.825731896" watchObservedRunningTime="2026-11-03 08:39:09.052429015 +0000 UTC m=+13.826049068"
Nov 03 08:39:09.287983 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:09.287915    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs" podStartSLOduration=4.035996212 podStartE2EDuration="13.287899624s" totalImagesPullingTime="9.251903412s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 08:38:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:09.287642041 +0000 UTC m=+14.061262095" watchObservedRunningTime="2026-11-03 08:39:09.287899624 +0000 UTC m=+14.061519676"
Nov 03 08:39:09.703342 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:09.703312    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:39:09.703506 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:09.703369    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pflts"
Nov 03 08:39:09.703506 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:09.703373    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-clx84"
Nov 03 08:39:09.703506 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:09.703461    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9jpw" podUID="906c75b6-325c-4e89-8a10-e4e355884133"
Nov 03 08:39:09.703880 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:09.703857    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pflts" podUID="7c26dfa4-e47e-43a1-94a2-816d22fb7f3a"
Nov 03 08:39:09.703969 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:09.703935    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-clx84" podUID="f6ca3dc3-231e-4a8d-a16c-7384c225110a"
Nov 03 08:39:10.835209 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:10.835173    2575 generic.go:356] "Generic (PLEG): container finished" podID="968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9" containerID="4a82dfa1b9438238ba347e874e04df130aebbbd52027093a07fb72f272483a27" exitCode=0
Nov 03 08:39:10.835739 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:10.835218    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wsrmr" event={"ID":"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9","Type":"ContainerDied","Data":"4a82dfa1b9438238ba347e874e04df130aebbbd52027093a07fb72f272483a27"}
Nov 03 08:39:11.703112 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:11.703067    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pflts"
Nov 03 08:39:11.703292 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:11.703067    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-clx84"
Nov 03 08:39:11.703292 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:11.703221    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pflts" podUID="7c26dfa4-e47e-43a1-94a2-816d22fb7f3a"
Nov 03 08:39:11.703415 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:11.703291    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-clx84" podUID="f6ca3dc3-231e-4a8d-a16c-7384c225110a"
Nov 03 08:39:11.703415 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:11.703083    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:39:11.703415 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:11.703384    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9jpw" podUID="906c75b6-325c-4e89-8a10-e4e355884133"
Nov 03 08:39:13.469444 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:13.469206    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/906c75b6-325c-4e89-8a10-e4e355884133-original-pull-secret\") pod \"global-pull-secret-syncer-x9jpw\" (UID: \"906c75b6-325c-4e89-8a10-e4e355884133\") " pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:39:13.469444 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:13.469420    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c26dfa4-e47e-43a1-94a2-816d22fb7f3a-metrics-certs\") pod \"network-metrics-daemon-pflts\" (UID: \"7c26dfa4-e47e-43a1-94a2-816d22fb7f3a\") " pod="openshift-multus/network-metrics-daemon-pflts"
Nov 03 08:39:13.470103 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:13.469340    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 08:39:13.470103 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:13.469520    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 08:39:13.470103 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:13.469534    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/906c75b6-325c-4e89-8a10-e4e355884133-original-pull-secret podName:906c75b6-325c-4e89-8a10-e4e355884133 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:29.469516265 +0000 UTC m=+34.243136309 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/906c75b6-325c-4e89-8a10-e4e355884133-original-pull-secret") pod "global-pull-secret-syncer-x9jpw" (UID: "906c75b6-325c-4e89-8a10-e4e355884133") : object "kube-system"/"original-pull-secret" not registered
Nov 03 08:39:13.470103 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:13.469561    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c26dfa4-e47e-43a1-94a2-816d22fb7f3a-metrics-certs podName:7c26dfa4-e47e-43a1-94a2-816d22fb7f3a nodeName:}" failed. No retries permitted until 2026-11-03 08:39:29.469546427 +0000 UTC m=+34.243166672 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c26dfa4-e47e-43a1-94a2-816d22fb7f3a-metrics-certs") pod "network-metrics-daemon-pflts" (UID: "7c26dfa4-e47e-43a1-94a2-816d22fb7f3a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 08:39:13.570145 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:13.570116    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcm5z\" (UniqueName: \"kubernetes.io/projected/f6ca3dc3-231e-4a8d-a16c-7384c225110a-kube-api-access-fcm5z\") pod \"network-check-target-clx84\" (UID: \"f6ca3dc3-231e-4a8d-a16c-7384c225110a\") " pod="openshift-network-diagnostics/network-check-target-clx84"
Nov 03 08:39:13.570276 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:13.570243    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 08:39:13.570276 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:13.570258    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 08:39:13.570276 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:13.570267    2575 projected.go:196] Error preparing data for projected volume kube-api-access-fcm5z for pod openshift-network-diagnostics/network-check-target-clx84: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 08:39:13.570366 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:13.570313    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f6ca3dc3-231e-4a8d-a16c-7384c225110a-kube-api-access-fcm5z podName:f6ca3dc3-231e-4a8d-a16c-7384c225110a nodeName:}" failed. No retries permitted until 2026-11-03 08:39:29.570297882 +0000 UTC m=+34.343917916 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fcm5z" (UniqueName: "kubernetes.io/projected/f6ca3dc3-231e-4a8d-a16c-7384c225110a-kube-api-access-fcm5z") pod "network-check-target-clx84" (UID: "f6ca3dc3-231e-4a8d-a16c-7384c225110a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 08:39:13.702877 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:13.702849    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pflts"
Nov 03 08:39:13.703020 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:13.702850    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-clx84"
Nov 03 08:39:13.703020 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:13.702963    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pflts" podUID="7c26dfa4-e47e-43a1-94a2-816d22fb7f3a"
Nov 03 08:39:13.703114 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:13.703038    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-clx84" podUID="f6ca3dc3-231e-4a8d-a16c-7384c225110a"
Nov 03 08:39:13.703114 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:13.702857    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:39:13.703114 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:13.703095    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9jpw" podUID="906c75b6-325c-4e89-8a10-e4e355884133"
Nov 03 08:39:13.847136 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:13.847074    2575 generic.go:356] "Generic (PLEG): container finished" podID="968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9" containerID="e4382c2a8bdc72875523687fcd7fc301f8420d7fa5157f0c93fdda3185e11a5a" exitCode=0
Nov 03 08:39:13.847283 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:13.847147    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wsrmr" event={"ID":"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9","Type":"ContainerDied","Data":"e4382c2a8bdc72875523687fcd7fc301f8420d7fa5157f0c93fdda3185e11a5a"}
Nov 03 08:39:13.849829 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:13.849811    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 08:39:13.851971 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:13.851837    2575 generic.go:356] "Generic (PLEG): container finished" podID="0a5d9c55-0799-4ae1-a902-dc06f813907c" containerID="ebce2c73bb3f0fe5d0fcbb1344c342a30e2bc04e87b103a31f437aa19b12a2af" exitCode=1
Nov 03 08:39:13.851971 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:13.851875    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt" event={"ID":"0a5d9c55-0799-4ae1-a902-dc06f813907c","Type":"ContainerStarted","Data":"de49915dbbb0ce8004f92a90a7a9d096e0b2338f4b48fb5b93aee6d3c2e2242a"}
Nov 03 08:39:13.851971 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:13.851918    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt" event={"ID":"0a5d9c55-0799-4ae1-a902-dc06f813907c","Type":"ContainerStarted","Data":"7471bb87c81d1eff94bb62fa2185fc35712c451e297d1e20d5e562773def0c2c"}
Nov 03 08:39:13.852070 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:13.851974    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt" event={"ID":"0a5d9c55-0799-4ae1-a902-dc06f813907c","Type":"ContainerStarted","Data":"eb51e62cfa907714ce609c86e18ec4a74895491c75c50d46bb9b7b1892bc0815"}
Nov 03 08:39:13.852070 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:13.852002    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt" event={"ID":"0a5d9c55-0799-4ae1-a902-dc06f813907c","Type":"ContainerStarted","Data":"668b41fa6c3fb7ed1ae0803fdac4597f224804d9c9bc74adba7232ed54477e40"}
Nov 03 08:39:13.852070 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:13.852037    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt" event={"ID":"0a5d9c55-0799-4ae1-a902-dc06f813907c","Type":"ContainerDied","Data":"ebce2c73bb3f0fe5d0fcbb1344c342a30e2bc04e87b103a31f437aa19b12a2af"}
Nov 03 08:39:13.852070 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:13.852064    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt" event={"ID":"0a5d9c55-0799-4ae1-a902-dc06f813907c","Type":"ContainerStarted","Data":"0b0662dec8e78df9346631cdcd78b32a519dcbab5920eeb5f76ae77b3c063c28"}
Nov 03 08:39:14.857116 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:14.857085    2575 generic.go:356] "Generic (PLEG): container finished" podID="968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9" containerID="7e09b678185b447d6f4012ef095b0b5e5ba6e5fac2b3697eff203580a4eed654" exitCode=0
Nov 03 08:39:14.857116 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:14.857120    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wsrmr" event={"ID":"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9","Type":"ContainerDied","Data":"7e09b678185b447d6f4012ef095b0b5e5ba6e5fac2b3697eff203580a4eed654"}
Nov 03 08:39:15.704033 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:15.703809    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pflts"
Nov 03 08:39:15.704160 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:15.703873    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-clx84"
Nov 03 08:39:15.704160 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:15.704061    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pflts" podUID="7c26dfa4-e47e-43a1-94a2-816d22fb7f3a"
Nov 03 08:39:15.704160 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:15.704127    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-clx84" podUID="f6ca3dc3-231e-4a8d-a16c-7384c225110a"
Nov 03 08:39:15.704160 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:15.703898    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:39:15.704272 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:15.704224    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9jpw" podUID="906c75b6-325c-4e89-8a10-e4e355884133"
Nov 03 08:39:15.862985 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:15.862938    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wsrmr" event={"ID":"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9","Type":"ContainerStarted","Data":"1c51e43dcdaacf16e64ea60748fd91c3134408f700320f05289219c6107f9d57"}
Nov 03 08:39:15.865887 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:15.865859    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 08:39:15.866212 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:15.866195    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt" event={"ID":"0a5d9c55-0799-4ae1-a902-dc06f813907c","Type":"ContainerStarted","Data":"f415d7dcae66cec672daa5bf535563982f1c40eb77aa048d9837626b5d5bf281"}
Nov 03 08:39:15.983518 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:15.983444    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wsrmr" podStartSLOduration=7.594136489 podStartE2EDuration="19.983431052s" totalImagesPullingTime="12.389294563s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 08:38:56 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:15.98320175 +0000 UTC m=+20.756821802" watchObservedRunningTime="2026-11-03 08:39:15.983431052 +0000 UTC m=+20.757051105"
Nov 03 08:39:17.702730 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:17.702700    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:39:17.702730 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:17.702720    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pflts"
Nov 03 08:39:17.703356 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:17.702832    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-clx84"
Nov 03 08:39:17.703356 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:17.702835    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9jpw" podUID="906c75b6-325c-4e89-8a10-e4e355884133"
Nov 03 08:39:17.703356 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:17.702956    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pflts" podUID="7c26dfa4-e47e-43a1-94a2-816d22fb7f3a"
Nov 03 08:39:17.703356 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:17.703053    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-clx84" podUID="f6ca3dc3-231e-4a8d-a16c-7384c225110a"
Nov 03 08:39:17.873173 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:17.873150    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 08:39:17.873540 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:17.873520    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt" event={"ID":"0a5d9c55-0799-4ae1-a902-dc06f813907c","Type":"ContainerStarted","Data":"b349a5c96962237e6fc963e8b79e4728bdc252b95dcd33172526735184f17d66"}
Nov 03 08:39:17.873734 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:17.873716    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:17.873900 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:17.873883    2575 scope.go:122] "RemoveContainer" containerID="ebce2c73bb3f0fe5d0fcbb1344c342a30e2bc04e87b103a31f437aa19b12a2af"
Nov 03 08:39:17.891604 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:17.891580    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:18.880007 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:18.879980    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 08:39:18.880487 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:18.880383    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt" event={"ID":"0a5d9c55-0799-4ae1-a902-dc06f813907c","Type":"ContainerStarted","Data":"6f3dd5f69b3baaf033eb44b67b1bc62adfe83060b5ace4cc0edd505d191723fa"}
Nov 03 08:39:18.880578 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:18.880564    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:18.880638 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:18.880628    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:18.896154 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:18.896130    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:19.052324 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:19.052279    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt" podStartSLOduration=8.121710553 podStartE2EDuration="23.052267123s" totalImagesPullingTime="14.93055657s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 08:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:19.051754845 +0000 UTC m=+23.825374897" watchObservedRunningTime="2026-11-03 08:39:19.052267123 +0000 UTC m=+23.825887212"
Nov 03 08:39:19.702672 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:19.702639    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:39:19.702672 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:19.702654    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pflts"
Nov 03 08:39:19.702891 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:19.702744    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9jpw" podUID="906c75b6-325c-4e89-8a10-e4e355884133"
Nov 03 08:39:19.702891 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:19.702777    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-clx84"
Nov 03 08:39:19.702891 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:19.702878    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-clx84" podUID="f6ca3dc3-231e-4a8d-a16c-7384c225110a"
Nov 03 08:39:19.703069 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:19.702923    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pflts" podUID="7c26dfa4-e47e-43a1-94a2-816d22fb7f3a"
Nov 03 08:39:20.270301 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:20.270077    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x9jpw"]
Nov 03 08:39:20.270722 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:20.270409    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:39:20.270722 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:20.270488    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9jpw" podUID="906c75b6-325c-4e89-8a10-e4e355884133"
Nov 03 08:39:20.272646 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:20.272614    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-clx84"]
Nov 03 08:39:20.272763 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:20.272726    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-clx84"
Nov 03 08:39:20.272846 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:20.272819    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-clx84" podUID="f6ca3dc3-231e-4a8d-a16c-7384c225110a"
Nov 03 08:39:20.273234 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:20.273214    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pflts"]
Nov 03 08:39:20.273339 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:20.273325    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pflts"
Nov 03 08:39:20.273449 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:20.273430    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pflts" podUID="7c26dfa4-e47e-43a1-94a2-816d22fb7f3a"
Nov 03 08:39:21.702776 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:21.702743    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-clx84"
Nov 03 08:39:21.703260 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:21.702803    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:39:21.703260 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:21.702749    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pflts"
Nov 03 08:39:21.703260 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:21.702869    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-clx84" podUID="f6ca3dc3-231e-4a8d-a16c-7384c225110a"
Nov 03 08:39:21.703260 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:21.702971    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pflts" podUID="7c26dfa4-e47e-43a1-94a2-816d22fb7f3a"
Nov 03 08:39:21.703260 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:21.703040    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9jpw" podUID="906c75b6-325c-4e89-8a10-e4e355884133"
Nov 03 08:39:23.703272 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:23.703237    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:39:23.703272 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:23.703250    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-clx84"
Nov 03 08:39:23.703706 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:23.703358    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9jpw" podUID="906c75b6-325c-4e89-8a10-e4e355884133"
Nov 03 08:39:23.703706 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:23.703389    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-clx84" podUID="f6ca3dc3-231e-4a8d-a16c-7384c225110a"
Nov 03 08:39:23.703706 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:23.703430    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pflts"
Nov 03 08:39:23.703706 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:23.703522    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pflts" podUID="7c26dfa4-e47e-43a1-94a2-816d22fb7f3a"
Nov 03 08:39:25.703630 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:25.703598    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pflts"
Nov 03 08:39:25.704144 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:25.703703    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:39:25.704144 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:25.703751    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-clx84"
Nov 03 08:39:25.704144 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:25.703812    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pflts" podUID="7c26dfa4-e47e-43a1-94a2-816d22fb7f3a"
Nov 03 08:39:25.704144 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:25.703830    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-clx84" podUID="f6ca3dc3-231e-4a8d-a16c-7384c225110a"
Nov 03 08:39:25.704144 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:25.703919    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9jpw" podUID="906c75b6-325c-4e89-8a10-e4e355884133"
Nov 03 08:39:26.016704 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.016678    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeReady"
Nov 03 08:39:26.016851 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.016795    2575 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 08:39:26.131423 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.131345    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gvbpd"]
Nov 03 08:39:26.134662 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.134636    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tft2z"]
Nov 03 08:39:26.134797 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.134701    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gvbpd"
Nov 03 08:39:26.138879 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.138859    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-74dwn"]
Nov 03 08:39:26.139031 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.139009    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tft2z"
Nov 03 08:39:26.142470 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.142135    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-74dwn"
Nov 03 08:39:26.181132 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.181111    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gvbpd"]
Nov 03 08:39:26.197644 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.197554    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 08:39:26.197644 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.197585    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sxt9c\""
Nov 03 08:39:26.197644 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.197620    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 08:39:26.197644 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.197626    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 08:39:26.197904 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.197710    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 08:39:26.197904 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.197775    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lngsh\""
Nov 03 08:39:26.198070 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.198048    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-74dwn"]
Nov 03 08:39:26.238590 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.238567    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tft2z"]
Nov 03 08:39:26.262331 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.262293    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-data-volume\") pod \"insights-runtime-extractor-tft2z\" (UID: \"49fbaf82-5d99-48e1-90eb-005aec9f3d1b\") " pod="openshift-insights/insights-runtime-extractor-tft2z"
Nov 03 08:39:26.262331 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.262336    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv94t\" (UniqueName: \"kubernetes.io/projected/d7329659-0fc1-4d97-b321-11e90bf7d13c-kube-api-access-kv94t\") pod \"ingress-canary-74dwn\" (UID: \"d7329659-0fc1-4d97-b321-11e90bf7d13c\") " pod="openshift-ingress-canary/ingress-canary-74dwn"
Nov 03 08:39:26.262510 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.262352    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-crio-socket\") pod \"insights-runtime-extractor-tft2z\" (UID: \"49fbaf82-5d99-48e1-90eb-005aec9f3d1b\") " pod="openshift-insights/insights-runtime-extractor-tft2z"
Nov 03 08:39:26.262510 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.262368    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tft2z\" (UID: \"49fbaf82-5d99-48e1-90eb-005aec9f3d1b\") " pod="openshift-insights/insights-runtime-extractor-tft2z"
Nov 03 08:39:26.262612 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.262527    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e8945a0-1bd9-4179-8996-a165bb1db1c4-tmp-dir\") pod \"dns-default-gvbpd\" (UID: \"9e8945a0-1bd9-4179-8996-a165bb1db1c4\") " pod="openshift-dns/dns-default-gvbpd"
Nov 03 08:39:26.262612 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.262582    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7329659-0fc1-4d97-b321-11e90bf7d13c-cert\") pod \"ingress-canary-74dwn\" (UID: \"d7329659-0fc1-4d97-b321-11e90bf7d13c\") " pod="openshift-ingress-canary/ingress-canary-74dwn"
Nov 03 08:39:26.262681 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.262638    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9e8945a0-1bd9-4179-8996-a165bb1db1c4-config-volume\") pod \"dns-default-gvbpd\" (UID: \"9e8945a0-1bd9-4179-8996-a165bb1db1c4\") " pod="openshift-dns/dns-default-gvbpd"
Nov 03 08:39:26.262681 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.262676    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swvhl\" (UniqueName: \"kubernetes.io/projected/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-kube-api-access-swvhl\") pod \"insights-runtime-extractor-tft2z\" (UID: \"49fbaf82-5d99-48e1-90eb-005aec9f3d1b\") " pod="openshift-insights/insights-runtime-extractor-tft2z"
Nov 03 08:39:26.262793 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.262775    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e8945a0-1bd9-4179-8996-a165bb1db1c4-metrics-tls\") pod \"dns-default-gvbpd\" (UID: \"9e8945a0-1bd9-4179-8996-a165bb1db1c4\") " pod="openshift-dns/dns-default-gvbpd"
Nov 03 08:39:26.262846 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.262806    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tft2z\" (UID: \"49fbaf82-5d99-48e1-90eb-005aec9f3d1b\") " pod="openshift-insights/insights-runtime-extractor-tft2z"
Nov 03 08:39:26.262846 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.262837    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nz7q\" (UniqueName: \"kubernetes.io/projected/9e8945a0-1bd9-4179-8996-a165bb1db1c4-kube-api-access-7nz7q\") pod \"dns-default-gvbpd\" (UID: \"9e8945a0-1bd9-4179-8996-a165bb1db1c4\") " pod="openshift-dns/dns-default-gvbpd"
Nov 03 08:39:26.299049 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.299025    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 08:39:26.299049 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.299046    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 08:39:26.299235 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.299219    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 08:39:26.299290 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.299275    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 08:39:26.299332 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.299292    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 08:39:26.299379 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.299348    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-2st7v\""
Nov 03 08:39:26.363201 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.363177    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-data-volume\") pod \"insights-runtime-extractor-tft2z\" (UID: \"49fbaf82-5d99-48e1-90eb-005aec9f3d1b\") " pod="openshift-insights/insights-runtime-extractor-tft2z"
Nov 03 08:39:26.363303 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.363206    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv94t\" (UniqueName: \"kubernetes.io/projected/d7329659-0fc1-4d97-b321-11e90bf7d13c-kube-api-access-kv94t\") pod \"ingress-canary-74dwn\" (UID: \"d7329659-0fc1-4d97-b321-11e90bf7d13c\") " pod="openshift-ingress-canary/ingress-canary-74dwn"
Nov 03 08:39:26.363303 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.363222    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-crio-socket\") pod \"insights-runtime-extractor-tft2z\" (UID: \"49fbaf82-5d99-48e1-90eb-005aec9f3d1b\") " pod="openshift-insights/insights-runtime-extractor-tft2z"
Nov 03 08:39:26.363303 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.363237    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tft2z\" (UID: \"49fbaf82-5d99-48e1-90eb-005aec9f3d1b\") " pod="openshift-insights/insights-runtime-extractor-tft2z"
Nov 03 08:39:26.363303 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.363301    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e8945a0-1bd9-4179-8996-a165bb1db1c4-tmp-dir\") pod \"dns-default-gvbpd\" (UID: \"9e8945a0-1bd9-4179-8996-a165bb1db1c4\") " pod="openshift-dns/dns-default-gvbpd"
Nov 03 08:39:26.363496 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.363341    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7329659-0fc1-4d97-b321-11e90bf7d13c-cert\") pod \"ingress-canary-74dwn\" (UID: \"d7329659-0fc1-4d97-b321-11e90bf7d13c\") " pod="openshift-ingress-canary/ingress-canary-74dwn"
Nov 03 08:39:26.363496 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.363365    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9e8945a0-1bd9-4179-8996-a165bb1db1c4-config-volume\") pod \"dns-default-gvbpd\" (UID: \"9e8945a0-1bd9-4179-8996-a165bb1db1c4\") " pod="openshift-dns/dns-default-gvbpd"
Nov 03 08:39:26.363496 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.363386    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swvhl\" (UniqueName: \"kubernetes.io/projected/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-kube-api-access-swvhl\") pod \"insights-runtime-extractor-tft2z\" (UID: \"49fbaf82-5d99-48e1-90eb-005aec9f3d1b\") " pod="openshift-insights/insights-runtime-extractor-tft2z"
Nov 03 08:39:26.363496 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:26.363401    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 08:39:26.363496 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.363429    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e8945a0-1bd9-4179-8996-a165bb1db1c4-metrics-tls\") pod \"dns-default-gvbpd\" (UID: \"9e8945a0-1bd9-4179-8996-a165bb1db1c4\") " pod="openshift-dns/dns-default-gvbpd"
Nov 03 08:39:26.363496 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.363435    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-crio-socket\") pod \"insights-runtime-extractor-tft2z\" (UID: \"49fbaf82-5d99-48e1-90eb-005aec9f3d1b\") " pod="openshift-insights/insights-runtime-extractor-tft2z"
Nov 03 08:39:26.363496 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.363452    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tft2z\" (UID: \"49fbaf82-5d99-48e1-90eb-005aec9f3d1b\") " pod="openshift-insights/insights-runtime-extractor-tft2z"
Nov 03 08:39:26.363496 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.363485    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nz7q\" (UniqueName: \"kubernetes.io/projected/9e8945a0-1bd9-4179-8996-a165bb1db1c4-kube-api-access-7nz7q\") pod \"dns-default-gvbpd\" (UID: \"9e8945a0-1bd9-4179-8996-a165bb1db1c4\") " pod="openshift-dns/dns-default-gvbpd"
Nov 03 08:39:26.363858 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:26.363566    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-insights-runtime-extractor-tls podName:49fbaf82-5d99-48e1-90eb-005aec9f3d1b nodeName:}" failed. No retries permitted until 2026-11-03 08:39:26.863546756 +0000 UTC m=+31.637166787 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tft2z" (UID: "49fbaf82-5d99-48e1-90eb-005aec9f3d1b") : secret "insights-runtime-extractor-tls" not found
Nov 03 08:39:26.363858 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.363657    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-data-volume\") pod \"insights-runtime-extractor-tft2z\" (UID: \"49fbaf82-5d99-48e1-90eb-005aec9f3d1b\") " pod="openshift-insights/insights-runtime-extractor-tft2z"
Nov 03 08:39:26.363858 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:26.363728    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 08:39:26.363858 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:26.363765    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 08:39:26.363858 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:26.363772    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-kube-rbac-proxy-cm podName:49fbaf82-5d99-48e1-90eb-005aec9f3d1b nodeName:}" failed. No retries permitted until 2026-11-03 08:39:26.863759817 +0000 UTC m=+31.637379874 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tft2z" (UID: "49fbaf82-5d99-48e1-90eb-005aec9f3d1b") : configmap "kube-rbac-proxy" not found
Nov 03 08:39:26.363858 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:26.363827    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e8945a0-1bd9-4179-8996-a165bb1db1c4-metrics-tls podName:9e8945a0-1bd9-4179-8996-a165bb1db1c4 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:26.86381588 +0000 UTC m=+31.637435911 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9e8945a0-1bd9-4179-8996-a165bb1db1c4-metrics-tls") pod "dns-default-gvbpd" (UID: "9e8945a0-1bd9-4179-8996-a165bb1db1c4") : secret "dns-default-metrics-tls" not found
Nov 03 08:39:26.363858 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.363852    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e8945a0-1bd9-4179-8996-a165bb1db1c4-tmp-dir\") pod \"dns-default-gvbpd\" (UID: \"9e8945a0-1bd9-4179-8996-a165bb1db1c4\") " pod="openshift-dns/dns-default-gvbpd"
Nov 03 08:39:26.364206 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.364057    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9e8945a0-1bd9-4179-8996-a165bb1db1c4-config-volume\") pod \"dns-default-gvbpd\" (UID: \"9e8945a0-1bd9-4179-8996-a165bb1db1c4\") " pod="openshift-dns/dns-default-gvbpd"
Nov 03 08:39:26.367659 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.367619    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7329659-0fc1-4d97-b321-11e90bf7d13c-cert\") pod \"ingress-canary-74dwn\" (UID: \"d7329659-0fc1-4d97-b321-11e90bf7d13c\") " pod="openshift-ingress-canary/ingress-canary-74dwn"
Nov 03 08:39:26.422033 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.421982    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swvhl\" (UniqueName: \"kubernetes.io/projected/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-kube-api-access-swvhl\") pod \"insights-runtime-extractor-tft2z\" (UID: \"49fbaf82-5d99-48e1-90eb-005aec9f3d1b\") " pod="openshift-insights/insights-runtime-extractor-tft2z"
Nov 03 08:39:26.422148 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.422130    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv94t\" (UniqueName: \"kubernetes.io/projected/d7329659-0fc1-4d97-b321-11e90bf7d13c-kube-api-access-kv94t\") pod \"ingress-canary-74dwn\" (UID: \"d7329659-0fc1-4d97-b321-11e90bf7d13c\") " pod="openshift-ingress-canary/ingress-canary-74dwn"
Nov 03 08:39:26.422528 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.422506    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nz7q\" (UniqueName: \"kubernetes.io/projected/9e8945a0-1bd9-4179-8996-a165bb1db1c4-kube-api-access-7nz7q\") pod \"dns-default-gvbpd\" (UID: \"9e8945a0-1bd9-4179-8996-a165bb1db1c4\") " pod="openshift-dns/dns-default-gvbpd"
Nov 03 08:39:26.463540 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.463521    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-74dwn"
Nov 03 08:39:26.849812 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.849735    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-74dwn"]
Nov 03 08:39:26.853594 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:39:26.853564    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7329659_0fc1_4d97_b321_11e90bf7d13c.slice/crio-2fb790962bf7280fc528b5620163aea464dd153e30db2c6bfb1081c97c98dd01 WatchSource:0}: Error finding container 2fb790962bf7280fc528b5620163aea464dd153e30db2c6bfb1081c97c98dd01: Status 404 returned error can't find the container with id 2fb790962bf7280fc528b5620163aea464dd153e30db2c6bfb1081c97c98dd01
Nov 03 08:39:26.868401 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.868380    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e8945a0-1bd9-4179-8996-a165bb1db1c4-metrics-tls\") pod \"dns-default-gvbpd\" (UID: \"9e8945a0-1bd9-4179-8996-a165bb1db1c4\") " pod="openshift-dns/dns-default-gvbpd"
Nov 03 08:39:26.868512 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.868413    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tft2z\" (UID: \"49fbaf82-5d99-48e1-90eb-005aec9f3d1b\") " pod="openshift-insights/insights-runtime-extractor-tft2z"
Nov 03 08:39:26.868512 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.868462    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tft2z\" (UID: \"49fbaf82-5d99-48e1-90eb-005aec9f3d1b\") " pod="openshift-insights/insights-runtime-extractor-tft2z"
Nov 03 08:39:26.868616 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:26.868530    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 08:39:26.868616 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:26.868562    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 08:39:26.868616 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:26.868572    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 08:39:26.868616 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:26.868588    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e8945a0-1bd9-4179-8996-a165bb1db1c4-metrics-tls podName:9e8945a0-1bd9-4179-8996-a165bb1db1c4 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:27.868573602 +0000 UTC m=+32.642193634 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9e8945a0-1bd9-4179-8996-a165bb1db1c4-metrics-tls") pod "dns-default-gvbpd" (UID: "9e8945a0-1bd9-4179-8996-a165bb1db1c4") : secret "dns-default-metrics-tls" not found
Nov 03 08:39:26.868616 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:26.868606    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-kube-rbac-proxy-cm podName:49fbaf82-5d99-48e1-90eb-005aec9f3d1b nodeName:}" failed. No retries permitted until 2026-11-03 08:39:27.868594701 +0000 UTC m=+32.642214744 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tft2z" (UID: "49fbaf82-5d99-48e1-90eb-005aec9f3d1b") : configmap "kube-rbac-proxy" not found
Nov 03 08:39:26.868793 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:26.868628    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-insights-runtime-extractor-tls podName:49fbaf82-5d99-48e1-90eb-005aec9f3d1b nodeName:}" failed. No retries permitted until 2026-11-03 08:39:27.868615156 +0000 UTC m=+32.642235215 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tft2z" (UID: "49fbaf82-5d99-48e1-90eb-005aec9f3d1b") : secret "insights-runtime-extractor-tls" not found
Nov 03 08:39:26.898993 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:26.898966    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-74dwn" event={"ID":"d7329659-0fc1-4d97-b321-11e90bf7d13c","Type":"ContainerStarted","Data":"2fb790962bf7280fc528b5620163aea464dd153e30db2c6bfb1081c97c98dd01"}
Nov 03 08:39:27.705963 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:27.705923    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-clx84"
Nov 03 08:39:27.706117 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:27.706011    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pflts"
Nov 03 08:39:27.706117 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:27.706100    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:39:27.760322 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:27.760301    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 08:39:27.760808 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:27.760790    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 08:39:27.760910 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:27.760837    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 08:39:27.810555 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:27.810513    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-htlrf\""
Nov 03 08:39:27.810694 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:27.810674    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 08:39:27.810694 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:27.810678    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jv5wj\""
Nov 03 08:39:27.810796 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:27.810685    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zgn57\""
Nov 03 08:39:27.876466 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:27.876413    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tft2z\" (UID: \"49fbaf82-5d99-48e1-90eb-005aec9f3d1b\") " pod="openshift-insights/insights-runtime-extractor-tft2z"
Nov 03 08:39:27.876885 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:27.876550    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e8945a0-1bd9-4179-8996-a165bb1db1c4-metrics-tls\") pod \"dns-default-gvbpd\" (UID: \"9e8945a0-1bd9-4179-8996-a165bb1db1c4\") " pod="openshift-dns/dns-default-gvbpd"
Nov 03 08:39:27.876885 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:27.876577    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tft2z\" (UID: \"49fbaf82-5d99-48e1-90eb-005aec9f3d1b\") " pod="openshift-insights/insights-runtime-extractor-tft2z"
Nov 03 08:39:27.877222 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:27.877202    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tft2z\" (UID: \"49fbaf82-5d99-48e1-90eb-005aec9f3d1b\") " pod="openshift-insights/insights-runtime-extractor-tft2z"
Nov 03 08:39:27.877340 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:27.877326    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 08:39:27.877411 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:27.877392    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-insights-runtime-extractor-tls podName:49fbaf82-5d99-48e1-90eb-005aec9f3d1b nodeName:}" failed. No retries permitted until 2026-11-03 08:39:29.877374404 +0000 UTC m=+34.650994449 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tft2z" (UID: "49fbaf82-5d99-48e1-90eb-005aec9f3d1b") : secret "insights-runtime-extractor-tls" not found
Nov 03 08:39:27.878073 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:27.878051    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 08:39:27.878176 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:27.878108    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e8945a0-1bd9-4179-8996-a165bb1db1c4-metrics-tls podName:9e8945a0-1bd9-4179-8996-a165bb1db1c4 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:29.878093288 +0000 UTC m=+34.651713320 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9e8945a0-1bd9-4179-8996-a165bb1db1c4-metrics-tls") pod "dns-default-gvbpd" (UID: "9e8945a0-1bd9-4179-8996-a165bb1db1c4") : secret "dns-default-metrics-tls" not found
Nov 03 08:39:29.483114 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:29.483084    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hgbb5_801881d6-f217-4c80-a190-af953df3af04/dns-node-resolver/0.log"
Nov 03 08:39:29.490374 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:29.490344    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/906c75b6-325c-4e89-8a10-e4e355884133-original-pull-secret\") pod \"global-pull-secret-syncer-x9jpw\" (UID: \"906c75b6-325c-4e89-8a10-e4e355884133\") " pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:39:29.490517 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:29.490428    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c26dfa4-e47e-43a1-94a2-816d22fb7f3a-metrics-certs\") pod \"network-metrics-daemon-pflts\" (UID: \"7c26dfa4-e47e-43a1-94a2-816d22fb7f3a\") " pod="openshift-multus/network-metrics-daemon-pflts"
Nov 03 08:39:29.490583 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:29.490566    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 08:39:29.490631 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:29.490621    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c26dfa4-e47e-43a1-94a2-816d22fb7f3a-metrics-certs podName:7c26dfa4-e47e-43a1-94a2-816d22fb7f3a nodeName:}" failed. No retries permitted until 2026-11-03 08:40:01.490608217 +0000 UTC m=+66.264228248 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c26dfa4-e47e-43a1-94a2-816d22fb7f3a-metrics-certs") pod "network-metrics-daemon-pflts" (UID: "7c26dfa4-e47e-43a1-94a2-816d22fb7f3a") : secret "metrics-daemon-secret" not found
Nov 03 08:39:29.499154 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:29.499125    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/906c75b6-325c-4e89-8a10-e4e355884133-original-pull-secret\") pod \"global-pull-secret-syncer-x9jpw\" (UID: \"906c75b6-325c-4e89-8a10-e4e355884133\") " pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:39:29.571585 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:29.571553    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:39:29.590896 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:29.590872    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcm5z\" (UniqueName: \"kubernetes.io/projected/f6ca3dc3-231e-4a8d-a16c-7384c225110a-kube-api-access-fcm5z\") pod \"network-check-target-clx84\" (UID: \"f6ca3dc3-231e-4a8d-a16c-7384c225110a\") " pod="openshift-network-diagnostics/network-check-target-clx84"
Nov 03 08:39:29.594101 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:29.594073    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcm5z\" (UniqueName: \"kubernetes.io/projected/f6ca3dc3-231e-4a8d-a16c-7384c225110a-kube-api-access-fcm5z\") pod \"network-check-target-clx84\" (UID: \"f6ca3dc3-231e-4a8d-a16c-7384c225110a\") " pod="openshift-network-diagnostics/network-check-target-clx84"
Nov 03 08:39:29.835617 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:29.835548    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-clx84"
Nov 03 08:39:29.893061 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:29.893027    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tft2z\" (UID: \"49fbaf82-5d99-48e1-90eb-005aec9f3d1b\") " pod="openshift-insights/insights-runtime-extractor-tft2z"
Nov 03 08:39:29.893188 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:29.893087    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e8945a0-1bd9-4179-8996-a165bb1db1c4-metrics-tls\") pod \"dns-default-gvbpd\" (UID: \"9e8945a0-1bd9-4179-8996-a165bb1db1c4\") " pod="openshift-dns/dns-default-gvbpd"
Nov 03 08:39:29.893188 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:29.893166    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 08:39:29.893188 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:29.893178    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 08:39:29.893285 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:29.893226    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-insights-runtime-extractor-tls podName:49fbaf82-5d99-48e1-90eb-005aec9f3d1b nodeName:}" failed. No retries permitted until 2026-11-03 08:39:33.89321192 +0000 UTC m=+38.666831952 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tft2z" (UID: "49fbaf82-5d99-48e1-90eb-005aec9f3d1b") : secret "insights-runtime-extractor-tls" not found
Nov 03 08:39:29.893285 ip-10-0-139-73 kubenswrapper[2575]: E1103 08:39:29.893243    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e8945a0-1bd9-4179-8996-a165bb1db1c4-metrics-tls podName:9e8945a0-1bd9-4179-8996-a165bb1db1c4 nodeName:}" failed. No retries permitted until 2026-11-03 08:39:33.893237085 +0000 UTC m=+38.666857115 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9e8945a0-1bd9-4179-8996-a165bb1db1c4-metrics-tls") pod "dns-default-gvbpd" (UID: "9e8945a0-1bd9-4179-8996-a165bb1db1c4") : secret "dns-default-metrics-tls" not found
Nov 03 08:39:29.979064 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:29.979034    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x9jpw"]
Nov 03 08:39:29.981922 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:39:29.981893    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod906c75b6_325c_4e89_8a10_e4e355884133.slice/crio-fe94beb078f134b2903a09f260a7c1330be776be4ae4520d84ffda96252c84ca WatchSource:0}: Error finding container fe94beb078f134b2903a09f260a7c1330be776be4ae4520d84ffda96252c84ca: Status 404 returned error can't find the container with id fe94beb078f134b2903a09f260a7c1330be776be4ae4520d84ffda96252c84ca
Nov 03 08:39:30.083701 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:30.083675    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5drgw_110651f3-9a70-43bf-888d-0813682bc891/node-ca/0.log"
Nov 03 08:39:30.269096 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:30.269062    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-clx84"]
Nov 03 08:39:30.272699 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:39:30.272671    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6ca3dc3_231e_4a8d_a16c_7384c225110a.slice/crio-814fc1234beb8c798254b95bb56f77b9ee9abee1c9b30e42e7aa077c3cc3f0e5 WatchSource:0}: Error finding container 814fc1234beb8c798254b95bb56f77b9ee9abee1c9b30e42e7aa077c3cc3f0e5: Status 404 returned error can't find the container with id 814fc1234beb8c798254b95bb56f77b9ee9abee1c9b30e42e7aa077c3cc3f0e5
Nov 03 08:39:30.910530 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:30.910499    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-74dwn" event={"ID":"d7329659-0fc1-4d97-b321-11e90bf7d13c","Type":"ContainerStarted","Data":"81f8a8962410eab538285f6c650b36c6adbd422e588c88491887cda0a6bfef04"}
Nov 03 08:39:30.911675 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:30.911642    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-clx84" event={"ID":"f6ca3dc3-231e-4a8d-a16c-7384c225110a","Type":"ContainerStarted","Data":"814fc1234beb8c798254b95bb56f77b9ee9abee1c9b30e42e7aa077c3cc3f0e5"}
Nov 03 08:39:30.912758 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:30.912735    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x9jpw" event={"ID":"906c75b6-325c-4e89-8a10-e4e355884133","Type":"ContainerStarted","Data":"fe94beb078f134b2903a09f260a7c1330be776be4ae4520d84ffda96252c84ca"}
Nov 03 08:39:31.026620 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:31.026567    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-74dwn" podStartSLOduration=1.970284486 podStartE2EDuration="5.026550206s" totalImagesPullingTime="3.05626572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:39:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:31.026374686 +0000 UTC m=+35.799994814" watchObservedRunningTime="2026-11-03 08:39:31.026550206 +0000 UTC m=+35.800170260"
Nov 03 08:39:31.083520 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:31.083488    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-74dwn_d7329659-0fc1-4d97-b321-11e90bf7d13c/serve-healthcheck-canary/0.log"
Nov 03 08:39:33.927846 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:33.927814    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tft2z\" (UID: \"49fbaf82-5d99-48e1-90eb-005aec9f3d1b\") " pod="openshift-insights/insights-runtime-extractor-tft2z"
Nov 03 08:39:33.928280 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:33.927888    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e8945a0-1bd9-4179-8996-a165bb1db1c4-metrics-tls\") pod \"dns-default-gvbpd\" (UID: \"9e8945a0-1bd9-4179-8996-a165bb1db1c4\") " pod="openshift-dns/dns-default-gvbpd"
Nov 03 08:39:33.930522 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:33.930497    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tft2z\" (UID: \"49fbaf82-5d99-48e1-90eb-005aec9f3d1b\") " pod="openshift-insights/insights-runtime-extractor-tft2z"
Nov 03 08:39:33.930522 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:33.930518    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e8945a0-1bd9-4179-8996-a165bb1db1c4-metrics-tls\") pod \"dns-default-gvbpd\" (UID: \"9e8945a0-1bd9-4179-8996-a165bb1db1c4\") " pod="openshift-dns/dns-default-gvbpd"
Nov 03 08:39:33.950614 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:33.950592    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gvbpd"
Nov 03 08:39:33.960746 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:33.960723    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tft2z"
Nov 03 08:39:34.551735 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:34.551661    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gvbpd"]
Nov 03 08:39:34.554728 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:39:34.554690    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e8945a0_1bd9_4179_8996_a165bb1db1c4.slice/crio-2839b31049e8b3e9fdc8ebada29c664da019834a925a4f9452314d958932725d WatchSource:0}: Error finding container 2839b31049e8b3e9fdc8ebada29c664da019834a925a4f9452314d958932725d: Status 404 returned error can't find the container with id 2839b31049e8b3e9fdc8ebada29c664da019834a925a4f9452314d958932725d
Nov 03 08:39:34.597314 ip-10-0-139-73 kubenswrapper[2575]: W1103 08:39:34.597288    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49fbaf82_5d99_48e1_90eb_005aec9f3d1b.slice/crio-83db3cb481cbf0cce124c9d83fecae81c881a9b35d4f3a701e460b34d7e2124f WatchSource:0}: Error finding container 83db3cb481cbf0cce124c9d83fecae81c881a9b35d4f3a701e460b34d7e2124f: Status 404 returned error can't find the container with id 83db3cb481cbf0cce124c9d83fecae81c881a9b35d4f3a701e460b34d7e2124f
Nov 03 08:39:34.602045 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:34.602019    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tft2z"]
Nov 03 08:39:34.924398 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:34.924261    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-clx84" event={"ID":"f6ca3dc3-231e-4a8d-a16c-7384c225110a","Type":"ContainerStarted","Data":"b16ea137e4a2bdd3ee28d68a2768c4217f66bc947ae76b7fc6aa4b4b48eb616e"}
Nov 03 08:39:34.924398 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:34.924335    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-clx84"
Nov 03 08:39:34.925812 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:34.925781    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gvbpd" event={"ID":"9e8945a0-1bd9-4179-8996-a165bb1db1c4","Type":"ContainerStarted","Data":"2839b31049e8b3e9fdc8ebada29c664da019834a925a4f9452314d958932725d"}
Nov 03 08:39:34.927479 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:34.927445    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tft2z" event={"ID":"49fbaf82-5d99-48e1-90eb-005aec9f3d1b","Type":"ContainerStarted","Data":"d7957ca3cd774144774a6092c6df1bb0c6fa4d508467c7ecdc0ad7568e2aefb1"}
Nov 03 08:39:34.927558 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:34.927499    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tft2z" event={"ID":"49fbaf82-5d99-48e1-90eb-005aec9f3d1b","Type":"ContainerStarted","Data":"83db3cb481cbf0cce124c9d83fecae81c881a9b35d4f3a701e460b34d7e2124f"}
Nov 03 08:39:35.039295 ip-10-0-139-73 kubenswrapper[2575]: I1103 08:39:35.039235    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-clx84" podStartSLOduration=35.16824485 podStartE2EDuration="39.039221222s" totalImagesPullingTime="3.870976372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:35.038907167 +0000 UTC m=+39.812527223" watchObservedRunningTime="2026-11-03 08:39:35.039221222 +0000 UTC m=+39.812841275"
Nov 03 08:39:35.668174 ip-10-0-139-73 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 08:39:35.675580 ip-10-0-139-73 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 08:39:35.675758 ip-10-0-139-73 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 08:39:35.675815 ip-10-0-139-73 systemd[1]: kubelet.service: Consumed 2.260s CPU time.
Nov 03 08:39:35.687469 ip-10-0-139-73 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 08:39:35.783484 ip-10-0-139-73 kubenswrapper[4473]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 08:39:35.783484 ip-10-0-139-73 kubenswrapper[4473]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 08:39:35.783484 ip-10-0-139-73 kubenswrapper[4473]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 08:39:35.783484 ip-10-0-139-73 kubenswrapper[4473]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 08:39:35.785550 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785470    4473 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 08:39:35.785550 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785550    4473 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 08:39:35.785550 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785553    4473 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 08:39:35.785647 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785558    4473 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 08:39:35.785647 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785560    4473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 08:39:35.785647 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785564    4473 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 08:39:35.785647 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785567    4473 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 08:39:35.785647 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785569    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 08:39:35.785647 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785579    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 08:39:35.785647 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785582    4473 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 08:39:35.785647 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785585    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 08:39:35.785647 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785588    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 08:39:35.785647 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785591    4473 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 08:39:35.785647 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785594    4473 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 08:39:35.785647 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785597    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 08:39:35.785647 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785599    4473 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 08:39:35.785647 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785602    4473 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 08:39:35.785647 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785606    4473 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 08:39:35.785647 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785609    4473 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 08:39:35.785647 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785612    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 08:39:35.785647 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785614    4473 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 08:39:35.785647 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785617    4473 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 08:39:35.786158 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785619    4473 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 08:39:35.786158 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785622    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 08:39:35.786158 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785625    4473 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 08:39:35.786158 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785627    4473 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 08:39:35.786158 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785630    4473 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 08:39:35.786158 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785632    4473 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 08:39:35.786158 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785634    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 08:39:35.786158 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785637    4473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 08:39:35.786158 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785639    4473 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 08:39:35.786158 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785642    4473 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 08:39:35.786158 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785644    4473 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 08:39:35.786158 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785647    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 08:39:35.786158 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785650    4473 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 08:39:35.786158 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785653    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 08:39:35.786158 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785656    4473 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 08:39:35.786158 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785659    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 08:39:35.786158 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785662    4473 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 08:39:35.786158 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785664    4473 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 08:39:35.786158 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785667    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 08:39:35.786610 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785669    4473 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 08:39:35.786610 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785672    4473 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 08:39:35.786610 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785675    4473 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 08:39:35.786610 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785678    4473 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 08:39:35.786610 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785681    4473 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 08:39:35.786610 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785683    4473 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 08:39:35.786610 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785686    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 08:39:35.786610 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785688    4473 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 08:39:35.786610 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785691    4473 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 08:39:35.786610 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785693    4473 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 08:39:35.786610 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785696    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 08:39:35.786610 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785698    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 08:39:35.786610 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785701    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 08:39:35.786610 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785704    4473 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 08:39:35.786610 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785707    4473 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 08:39:35.786610 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785710    4473 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 08:39:35.786610 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785713    4473 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 08:39:35.786610 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785715    4473 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 08:39:35.786610 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785718    4473 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 08:39:35.786610 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785720    4473 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 08:39:35.787116 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785723    4473 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 08:39:35.787116 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785725    4473 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 08:39:35.787116 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785728    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 08:39:35.787116 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785730    4473 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 08:39:35.787116 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785733    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 08:39:35.787116 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785736    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 08:39:35.787116 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785739    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 08:39:35.787116 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785742    4473 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 08:39:35.787116 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785744    4473 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 08:39:35.787116 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785747    4473 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 08:39:35.787116 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785750    4473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 08:39:35.787116 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785752    4473 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 08:39:35.787116 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785755    4473 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 08:39:35.787116 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785757    4473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 08:39:35.787116 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785760    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 08:39:35.787116 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785762    4473 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 08:39:35.787116 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785765    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 08:39:35.787116 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785767    4473 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 08:39:35.787116 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785770    4473 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 08:39:35.787583 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785772    4473 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 08:39:35.787583 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785775    4473 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 08:39:35.787583 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785777    4473 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 08:39:35.787583 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785780    4473 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 08:39:35.787583 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785784    4473 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 08:39:35.787583 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785788    4473 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 08:39:35.787583 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785790    4473 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 08:39:35.787583 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785793    4473 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 08:39:35.787583 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785796    4473 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 08:39:35.787583 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785798    4473 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 08:39:35.787583 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785801    4473 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 08:39:35.787583 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785804    4473 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 08:39:35.787583 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785806    4473 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 08:39:35.787583 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785809    4473 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 08:39:35.787583 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785812    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 08:39:35.787583 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785814    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 08:39:35.787583 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785817    4473 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 08:39:35.787583 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785819    4473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 08:39:35.787583 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785822    4473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 08:39:35.787583 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785824    4473 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 08:39:35.788074 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785828    4473 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 08:39:35.788074 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785831    4473 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 08:39:35.788074 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785833    4473 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 08:39:35.788074 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785836    4473 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 08:39:35.788074 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785838    4473 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 08:39:35.788074 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.785841    4473 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 08:39:35.788074 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786653    4473 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 08:39:35.788074 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786661    4473 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 08:39:35.788074 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786664    4473 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 08:39:35.788074 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786668    4473 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 08:39:35.788074 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786671    4473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 08:39:35.788074 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786674    4473 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 08:39:35.788074 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786676    4473 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 08:39:35.788074 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786679    4473 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 08:39:35.788074 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786682    4473 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 08:39:35.788074 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786684    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 08:39:35.788074 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786686    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 08:39:35.788074 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786689    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 08:39:35.788074 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786691    4473 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 08:39:35.788074 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786694    4473 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 08:39:35.788561 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786697    4473 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 08:39:35.788561 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786700    4473 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 08:39:35.788561 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786703    4473 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 08:39:35.788561 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786705    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 08:39:35.788561 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786708    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 08:39:35.788561 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786711    4473 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 08:39:35.788561 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786713    4473 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 08:39:35.788561 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786716    4473 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 08:39:35.788561 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786719    4473 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 08:39:35.788561 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786721    4473 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 08:39:35.788561 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786724    4473 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 08:39:35.788561 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786726    4473 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 08:39:35.788561 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786729    4473 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 08:39:35.788561 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786731    4473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 08:39:35.788561 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786735    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 08:39:35.788561 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786737    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 08:39:35.788561 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786739    4473 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 08:39:35.788561 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786742    4473 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 08:39:35.788561 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786744    4473 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 08:39:35.788561 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786747    4473 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 08:39:35.789085 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786749    4473 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 08:39:35.789085 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786752    4473 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 08:39:35.789085 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786754    4473 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 08:39:35.789085 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786757    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 08:39:35.789085 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786760    4473 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 08:39:35.789085 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786762    4473 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 08:39:35.789085 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786765    4473 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 08:39:35.789085 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786768    4473 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 08:39:35.789085 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786770    4473 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 08:39:35.789085 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786773    4473 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 08:39:35.789085 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786776    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 08:39:35.789085 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786779    4473 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 08:39:35.789085 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786781    4473 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 08:39:35.789085 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786784    4473 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 08:39:35.789085 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786786    4473 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 08:39:35.789085 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786789    4473 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 08:39:35.789085 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786791    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 08:39:35.789085 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786793    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 08:39:35.789085 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786796    4473 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 08:39:35.789085 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786798    4473 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 08:39:35.789566 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786801    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 08:39:35.789566 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786804    4473 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 08:39:35.789566 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786806    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 08:39:35.789566 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786808    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 08:39:35.789566 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786811    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 08:39:35.789566 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786813    4473 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 08:39:35.789566 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786816    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 08:39:35.789566 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786819    4473 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 08:39:35.789566 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786821    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 08:39:35.789566 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786824    4473 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 08:39:35.789566 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786826    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 08:39:35.789566 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786829    4473 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 08:39:35.789566 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786832    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 08:39:35.789566 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786835    4473 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 08:39:35.789566 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786838    4473 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 08:39:35.789566 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786840    4473 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 08:39:35.789566 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786843    4473 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 08:39:35.789566 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786845    4473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 08:39:35.789566 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786848    4473 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 08:39:35.790060 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786850    4473 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 08:39:35.790060 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786853    4473 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 08:39:35.790060 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786855    4473 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 08:39:35.790060 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786858    4473 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 08:39:35.790060 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786861    4473 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 08:39:35.790060 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786863    4473 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 08:39:35.790060 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786866    4473 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 08:39:35.790060 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786868    4473 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 08:39:35.790060 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786870    4473 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 08:39:35.790060 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786873    4473 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 08:39:35.790060 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786875    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 08:39:35.790060 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786878    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 08:39:35.790060 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786880    4473 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 08:39:35.790060 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786884    4473 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 08:39:35.790060 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786888    4473 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 08:39:35.790060 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786891    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 08:39:35.790060 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786893    4473 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 08:39:35.790060 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786896    4473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 08:39:35.790060 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786898    4473 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 08:39:35.790528 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786901    4473 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 08:39:35.790528 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786905    4473 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 08:39:35.790528 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786914    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 08:39:35.790528 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786917    4473 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 08:39:35.790528 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786921    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 08:39:35.790528 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786924    4473 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 08:39:35.790528 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786926    4473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 08:39:35.790528 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786929    4473 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 08:39:35.790528 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786931    4473 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 08:39:35.790528 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786934    4473 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 08:39:35.790528 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786937    4473 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 08:39:35.790528 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786939    4473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 08:39:35.790528 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786967    4473 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 08:39:35.790528 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.786972    4473 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 08:39:35.790528 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787490    4473 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 08:39:35.790528 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787494    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 08:39:35.790528 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787497    4473 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 08:39:35.790528 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787500    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 08:39:35.790528 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787502    4473 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 08:39:35.791121 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787505    4473 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 08:39:35.791121 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787507    4473 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 08:39:35.791121 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787510    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 08:39:35.791121 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787513    4473 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 08:39:35.791121 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787515    4473 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 08:39:35.791121 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787518    4473 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 08:39:35.791121 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787520    4473 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 08:39:35.791121 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787523    4473 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 08:39:35.791121 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787525    4473 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 08:39:35.791121 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787528    4473 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 08:39:35.791121 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787531    4473 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 08:39:35.791121 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787533    4473 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 08:39:35.791121 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787535    4473 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 08:39:35.791121 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787538    4473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 08:39:35.791121 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787542    4473 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 08:39:35.791121 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787545    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 08:39:35.791121 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787548    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 08:39:35.791121 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787550    4473 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 08:39:35.791121 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787553    4473 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 08:39:35.791121 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787556    4473 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 08:39:35.791606 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787558    4473 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 08:39:35.791606 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787561    4473 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 08:39:35.791606 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787563    4473 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 08:39:35.791606 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787567    4473 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 08:39:35.791606 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787570    4473 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 08:39:35.791606 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787573    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 08:39:35.791606 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787576    4473 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 08:39:35.791606 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787579    4473 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 08:39:35.791606 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787581    4473 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 08:39:35.791606 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787584    4473 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 08:39:35.791606 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787586    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 08:39:35.791606 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787589    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 08:39:35.791606 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787592    4473 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 08:39:35.791606 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787594    4473 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 08:39:35.791606 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787597    4473 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 08:39:35.791606 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787599    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 08:39:35.791606 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787602    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 08:39:35.791606 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787604    4473 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 08:39:35.791606 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787607    4473 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 08:39:35.792389 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787609    4473 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 08:39:35.792389 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787612    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 08:39:35.792389 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787615    4473 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 08:39:35.792389 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787618    4473 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 08:39:35.792389 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787620    4473 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 08:39:35.792389 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787623    4473 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 08:39:35.792389 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787626    4473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 08:39:35.792389 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787628    4473 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 08:39:35.792389 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787631    4473 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 08:39:35.792389 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787634    4473 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 08:39:35.792389 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787637    4473 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 08:39:35.792389 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787639    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 08:39:35.792389 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787642    4473 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 08:39:35.792389 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787644    4473 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 08:39:35.792389 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787647    4473 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 08:39:35.792389 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787649    4473 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 08:39:35.792389 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787651    4473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 08:39:35.792389 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787654    4473 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 08:39:35.792389 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787659    4473 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 08:39:35.792389 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787662    4473 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 08:39:35.792977 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787664    4473 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 08:39:35.792977 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787666    4473 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 08:39:35.792977 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787669    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 08:39:35.792977 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787671    4473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 08:39:35.792977 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787674    4473 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 08:39:35.792977 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787676    4473 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 08:39:35.792977 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787679    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 08:39:35.792977 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787681    4473 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 08:39:35.792977 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787684    4473 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 08:39:35.792977 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787686    4473 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 08:39:35.792977 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787692    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 08:39:35.792977 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787694    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 08:39:35.792977 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787697    4473 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 08:39:35.792977 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787701    4473 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 08:39:35.792977 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787704    4473 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 08:39:35.792977 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787707    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 08:39:35.792977 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787710    4473 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 08:39:35.792977 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787712    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 08:39:35.792977 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787715    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 08:39:35.793485 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787717    4473 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 08:39:35.793485 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787720    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 08:39:35.793485 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787722    4473 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 08:39:35.793485 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787731    4473 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 08:39:35.793485 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787734    4473 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 08:39:35.793485 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787736    4473 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 08:39:35.793485 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787739    4473 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 08:39:35.793485 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787741    4473 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 08:39:35.793485 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787744    4473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 08:39:35.793485 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787746    4473 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 08:39:35.793485 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787748    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 08:39:35.793485 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787752    4473 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 08:39:35.793485 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787754    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 08:39:35.793485 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787757    4473 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 08:39:35.793485 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787760    4473 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 08:39:35.793485 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787763    4473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 08:39:35.793485 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787766    4473 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 08:39:35.793485 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787768    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 08:39:35.793485 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787770    4473 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 08:39:35.793485 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787773    4473 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 08:39:35.794040 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787775    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 08:39:35.794040 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787778    4473 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 08:39:35.794040 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.787780    4473 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 08:39:35.794040 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.787881    4473 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 08:39:35.794040 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.787891    4473 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 08:39:35.794040 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.787906    4473 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 08:39:35.794040 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.787911    4473 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 08:39:35.794040 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.787915    4473 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 08:39:35.794040 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.787919    4473 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 08:39:35.794040 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.787923    4473 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 08:39:35.794040 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.787927    4473 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 08:39:35.794040 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.787930    4473 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 08:39:35.794040 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.787933    4473 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 08:39:35.794040 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.787936    4473 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 08:39:35.794040 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.787956    4473 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 08:39:35.794040 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.787960    4473 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 08:39:35.794040 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.787970    4473 flags.go:64] FLAG: --cgroup-root=""
Nov 03 08:39:35.794040 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.787973    4473 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 08:39:35.794040 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.787976    4473 flags.go:64] FLAG: --client-ca-file=""
Nov 03 08:39:35.794040 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.787979    4473 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 08:39:35.794040 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.787982    4473 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 08:39:35.794040 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.787994    4473 flags.go:64] FLAG: --cluster-domain=""
Nov 03 08:39:35.794040 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.787996    4473 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 08:39:35.794040 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788001    4473 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 08:39:35.794604 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788004    4473 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 08:39:35.794604 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788007    4473 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 08:39:35.794604 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788011    4473 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 08:39:35.794604 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788014    4473 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 08:39:35.794604 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788017    4473 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 08:39:35.794604 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788020    4473 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 08:39:35.794604 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788023    4473 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 08:39:35.794604 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788026    4473 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 08:39:35.794604 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788029    4473 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 08:39:35.794604 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788032    4473 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 08:39:35.794604 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788035    4473 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 08:39:35.794604 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788040    4473 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 08:39:35.794604 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788044    4473 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 08:39:35.794604 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788046    4473 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 08:39:35.794604 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788049    4473 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 08:39:35.794604 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788052    4473 flags.go:64] FLAG: --enable-server="true"
Nov 03 08:39:35.794604 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788065    4473 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 08:39:35.794604 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788073    4473 flags.go:64] FLAG: --event-burst="100"
Nov 03 08:39:35.794604 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788077    4473 flags.go:64] FLAG: --event-qps="50"
Nov 03 08:39:35.794604 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788080    4473 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 08:39:35.794604 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788083    4473 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 08:39:35.794604 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788086    4473 flags.go:64] FLAG: --eviction-hard=""
Nov 03 08:39:35.794604 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788091    4473 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 08:39:35.794604 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788094    4473 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 08:39:35.795207 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788097    4473 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 08:39:35.795207 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788105    4473 flags.go:64] FLAG: --eviction-soft=""
Nov 03 08:39:35.795207 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788109    4473 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 08:39:35.795207 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788112    4473 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 08:39:35.795207 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788115    4473 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 08:39:35.795207 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788118    4473 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 08:39:35.795207 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788121    4473 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 08:39:35.795207 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788125    4473 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 08:39:35.795207 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788127    4473 flags.go:64] FLAG: --feature-gates=""
Nov 03 08:39:35.795207 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788134    4473 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 08:39:35.795207 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788137    4473 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 08:39:35.795207 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788140    4473 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 08:39:35.795207 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788143    4473 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 08:39:35.795207 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788146    4473 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 08:39:35.795207 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788149    4473 flags.go:64] FLAG: --help="false"
Nov 03 08:39:35.795207 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788152    4473 flags.go:64] FLAG: --hostname-override="ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:39:35.795207 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788155    4473 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 08:39:35.795207 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788158    4473 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 08:39:35.795207 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788161    4473 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 08:39:35.795207 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788165    4473 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 08:39:35.795207 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788169    4473 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 08:39:35.795207 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788171    4473 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 08:39:35.795207 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788174    4473 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 08:39:35.795885 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788177    4473 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 08:39:35.795885 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788180    4473 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 08:39:35.795885 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788182    4473 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 08:39:35.795885 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788185    4473 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 08:39:35.795885 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788188    4473 flags.go:64] FLAG: --kube-reserved=""
Nov 03 08:39:35.795885 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788191    4473 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 08:39:35.795885 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788194    4473 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 08:39:35.795885 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788197    4473 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 08:39:35.795885 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788200    4473 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 08:39:35.795885 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788202    4473 flags.go:64] FLAG: --lock-file=""
Nov 03 08:39:35.795885 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788205    4473 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 08:39:35.795885 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788214    4473 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 08:39:35.795885 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788217    4473 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 08:39:35.795885 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788223    4473 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 08:39:35.795885 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788225    4473 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 08:39:35.795885 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788228    4473 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 08:39:35.795885 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788233    4473 flags.go:64] FLAG: --logging-format="text"
Nov 03 08:39:35.795885 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788236    4473 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 08:39:35.795885 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788239    4473 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 08:39:35.795885 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788242    4473 flags.go:64] FLAG: --manifest-url=""
Nov 03 08:39:35.795885 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788245    4473 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 08:39:35.795885 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788249    4473 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 08:39:35.795885 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788252    4473 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 08:39:35.795885 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788256    4473 flags.go:64] FLAG: --max-pods="110"
Nov 03 08:39:35.795885 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788259    4473 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 08:39:35.796514 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788261    4473 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 08:39:35.796514 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788264    4473 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 08:39:35.796514 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788267    4473 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 08:39:35.796514 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788271    4473 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 08:39:35.796514 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788274    4473 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 08:39:35.796514 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788277    4473 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 08:39:35.796514 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788283    4473 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 08:39:35.796514 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788286    4473 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 08:39:35.796514 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788289    4473 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 08:39:35.796514 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788293    4473 flags.go:64] FLAG: --pod-cidr=""
Nov 03 08:39:35.796514 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788295    4473 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 08:39:35.796514 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788298    4473 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 08:39:35.796514 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788301    4473 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 08:39:35.796514 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788304    4473 flags.go:64] FLAG: --port="10250"
Nov 03 08:39:35.796514 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788307    4473 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 08:39:35.796514 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788310    4473 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-025d50e6e4ea3a220"
Nov 03 08:39:35.796514 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788313    4473 flags.go:64] FLAG: --qos-reserved=""
Nov 03 08:39:35.796514 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788316    4473 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 08:39:35.796514 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788319    4473 flags.go:64] FLAG: --register-node="true"
Nov 03 08:39:35.796514 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788324    4473 flags.go:64] FLAG: --register-with-taints=""
Nov 03 08:39:35.796514 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788327    4473 flags.go:64] FLAG: --registry-burst="10"
Nov 03 08:39:35.796514 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788330    4473 flags.go:64] FLAG: --registry-qps="5"
Nov 03 08:39:35.796514 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788333    4473 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 08:39:35.796514 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788337    4473 flags.go:64] FLAG: --reserved-memory=""
Nov 03 08:39:35.796514 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788340    4473 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 08:39:35.796514 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788343    4473 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 08:39:35.797208 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788346    4473 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 08:39:35.797208 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788349    4473 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 08:39:35.797208 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788352    4473 flags.go:64] FLAG: --runonce="false"
Nov 03 08:39:35.797208 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788355    4473 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 08:39:35.797208 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788358    4473 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 08:39:35.797208 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788361    4473 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 08:39:35.797208 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788364    4473 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 08:39:35.797208 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788366    4473 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 08:39:35.797208 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788369    4473 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 08:39:35.797208 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788374    4473 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 08:39:35.797208 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788377    4473 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 08:39:35.797208 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788380    4473 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 08:39:35.797208 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788382    4473 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 08:39:35.797208 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788385    4473 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 08:39:35.797208 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788388    4473 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 08:39:35.797208 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788391    4473 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 08:39:35.797208 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788393    4473 flags.go:64] FLAG: --system-cgroups=""
Nov 03 08:39:35.797208 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788396    4473 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 08:39:35.797208 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788401    4473 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 08:39:35.797208 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788404    4473 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 08:39:35.797208 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788407    4473 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 08:39:35.797208 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788413    4473 flags.go:64] FLAG: --tls-min-version=""
Nov 03 08:39:35.797208 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788416    4473 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 08:39:35.797208 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788419    4473 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 08:39:35.797208 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788422    4473 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 08:39:35.797813 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788424    4473 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 08:39:35.797813 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788428    4473 flags.go:64] FLAG: --v="2"
Nov 03 08:39:35.797813 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788432    4473 flags.go:64] FLAG: --version="false"
Nov 03 08:39:35.797813 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788436    4473 flags.go:64] FLAG: --vmodule=""
Nov 03 08:39:35.797813 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788441    4473 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 08:39:35.797813 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788445    4473 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 08:39:35.797813 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788585    4473 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 08:39:35.797813 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788589    4473 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 08:39:35.797813 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788593    4473 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 08:39:35.797813 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788596    4473 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 08:39:35.797813 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788598    4473 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 08:39:35.797813 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788601    4473 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 08:39:35.797813 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788604    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 08:39:35.797813 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788606    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 08:39:35.797813 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788609    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 08:39:35.797813 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788611    4473 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 08:39:35.797813 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788614    4473 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 08:39:35.797813 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788617    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 08:39:35.797813 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788621    4473 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 08:39:35.797813 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788624    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 08:39:35.797813 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788626    4473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 08:39:35.798467 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788629    4473 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 08:39:35.798467 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788631    4473 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 08:39:35.798467 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788634    4473 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 08:39:35.798467 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788636    4473 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 08:39:35.798467 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788648    4473 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 08:39:35.798467 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788651    4473 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 08:39:35.798467 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788654    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 08:39:35.798467 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788656    4473 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 08:39:35.798467 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788659    4473 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 08:39:35.798467 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788661    4473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 08:39:35.798467 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788664    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 08:39:35.798467 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788667    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 08:39:35.798467 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788669    4473 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 08:39:35.798467 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788672    4473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 08:39:35.798467 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788675    4473 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 08:39:35.798467 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788679    4473 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 08:39:35.798467 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788682    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 08:39:35.798467 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788684    4473 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 08:39:35.798467 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788687    4473 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 08:39:35.798467 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788689    4473 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 08:39:35.798960 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788692    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 08:39:35.798960 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788694    4473 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 08:39:35.798960 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788696    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 08:39:35.798960 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788699    4473 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 08:39:35.798960 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788701    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 08:39:35.798960 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788704    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 08:39:35.798960 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788707    4473 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 08:39:35.798960 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788709    4473 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 08:39:35.798960 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788711    4473 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 08:39:35.798960 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788714    4473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 08:39:35.798960 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788716    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 08:39:35.798960 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788719    4473 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 08:39:35.798960 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788721    4473 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 08:39:35.798960 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788724    4473 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 08:39:35.798960 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788726    4473 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 08:39:35.798960 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788729    4473 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 08:39:35.798960 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788731    4473 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 08:39:35.798960 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788733    4473 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 08:39:35.798960 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788736    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 08:39:35.799462 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788738    4473 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 08:39:35.799462 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788741    4473 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 08:39:35.799462 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788743    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 08:39:35.799462 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788746    4473 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 08:39:35.799462 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788748    4473 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 08:39:35.799462 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788750    4473 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 08:39:35.799462 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788753    4473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 08:39:35.799462 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788756    4473 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 08:39:35.799462 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788760    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 08:39:35.799462 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788762    4473 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 08:39:35.799462 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788765    4473 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 08:39:35.799462 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788767    4473 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 08:39:35.799462 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788770    4473 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 08:39:35.799462 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788774    4473 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 08:39:35.799462 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788777    4473 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 08:39:35.799462 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788781    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 08:39:35.799462 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788785    4473 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 08:39:35.799462 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788787    4473 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 08:39:35.799462 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788790    4473 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 08:39:35.799924 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788793    4473 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 08:39:35.799924 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788795    4473 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 08:39:35.799924 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788798    4473 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 08:39:35.799924 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788800    4473 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 08:39:35.799924 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788803    4473 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 08:39:35.799924 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788805    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 08:39:35.799924 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788807    4473 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 08:39:35.799924 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788810    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 08:39:35.799924 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788812    4473 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 08:39:35.799924 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788815    4473 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 08:39:35.799924 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788817    4473 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 08:39:35.799924 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788819    4473 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 08:39:35.799924 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788822    4473 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 08:39:35.799924 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788824    4473 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 08:39:35.799924 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788827    4473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 08:39:35.799924 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788829    4473 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 08:39:35.799924 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788832    4473 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 08:39:35.799924 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788834    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 08:39:35.799924 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788836    4473 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 08:39:35.799924 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788839    4473 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 08:39:35.800507 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788841    4473 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 08:39:35.800507 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788848    4473 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 08:39:35.800507 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788851    4473 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 08:39:35.800507 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788853    4473 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 08:39:35.800507 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788856    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 08:39:35.800507 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788858    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 08:39:35.800507 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788860    4473 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 08:39:35.800507 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788863    4473 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 08:39:35.800507 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788865    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 08:39:35.800507 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788868    4473 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 08:39:35.800507 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788871    4473 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 08:39:35.800507 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788874    4473 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 08:39:35.800507 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788876    4473 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 08:39:35.800507 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.788902    4473 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 08:39:35.800507 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788938    4473 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 08:39:35.800507 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788960    4473 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 08:39:35.801010 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788963    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 08:39:35.801010 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788967    4473 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 08:39:35.801010 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788971    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 08:39:35.801010 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788974    4473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 08:39:35.801010 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788976    4473 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 08:39:35.801010 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788979    4473 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 08:39:35.801010 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788981    4473 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 08:39:35.801010 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788984    4473 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 08:39:35.801010 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788986    4473 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 08:39:35.801010 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788989    4473 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 08:39:35.801010 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788991    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 08:39:35.801010 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788993    4473 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 08:39:35.801010 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788996    4473 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 08:39:35.801010 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.788998    4473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 08:39:35.801010 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789001    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 08:39:35.801010 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789003    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 08:39:35.801010 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789005    4473 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 08:39:35.801010 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789009    4473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 08:39:35.801010 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789012    4473 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 08:39:35.801010 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789016    4473 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 08:39:35.801539 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789019    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 08:39:35.801539 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789021    4473 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 08:39:35.801539 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789024    4473 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 08:39:35.801539 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789026    4473 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 08:39:35.801539 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789029    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 08:39:35.801539 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789031    4473 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 08:39:35.801539 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789034    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 08:39:35.801539 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789036    4473 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 08:39:35.801539 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789039    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 08:39:35.801539 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789041    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 08:39:35.801539 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789044    4473 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 08:39:35.801539 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789047    4473 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 08:39:35.801539 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789049    4473 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 08:39:35.801539 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789052    4473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 08:39:35.801539 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789054    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 08:39:35.801539 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789057    4473 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 08:39:35.801539 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789059    4473 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 08:39:35.801539 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789062    4473 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 08:39:35.801539 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789064    4473 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 08:39:35.801539 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789067    4473 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 08:39:35.802061 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789069    4473 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 08:39:35.802061 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789071    4473 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 08:39:35.802061 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789074    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 08:39:35.802061 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789076    4473 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 08:39:35.802061 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789078    4473 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 08:39:35.802061 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789081    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 08:39:35.802061 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789084    4473 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 08:39:35.802061 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789086    4473 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 08:39:35.802061 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789089    4473 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 08:39:35.802061 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789093    4473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 08:39:35.802061 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789095    4473 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 08:39:35.802061 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789100    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 08:39:35.802061 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789102    4473 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 08:39:35.802061 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789105    4473 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 08:39:35.802061 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789108    4473 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 08:39:35.802061 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789110    4473 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 08:39:35.802061 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789114    4473 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 08:39:35.802061 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789117    4473 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 08:39:35.802061 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789121    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 08:39:35.802607 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789124    4473 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 08:39:35.802607 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789126    4473 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 08:39:35.802607 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789129    4473 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 08:39:35.802607 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789131    4473 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 08:39:35.802607 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789134    4473 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 08:39:35.802607 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789136    4473 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 08:39:35.802607 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789139    4473 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 08:39:35.802607 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789141    4473 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 08:39:35.802607 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789144    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 08:39:35.802607 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789146    4473 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 08:39:35.802607 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789149    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 08:39:35.802607 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789151    4473 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 08:39:35.802607 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789154    4473 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 08:39:35.802607 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789156    4473 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 08:39:35.802607 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789159    4473 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 08:39:35.802607 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789161    4473 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 08:39:35.802607 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789164    4473 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 08:39:35.802607 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789166    4473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 08:39:35.802607 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789169    4473 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 08:39:35.802607 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789171    4473 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 08:39:35.803264 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789175    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 08:39:35.803264 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789177    4473 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 08:39:35.803264 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789181    4473 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 08:39:35.803264 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789183    4473 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 08:39:35.803264 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789187    4473 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 08:39:35.803264 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789190    4473 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 08:39:35.803264 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789193    4473 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 08:39:35.803264 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789195    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 08:39:35.803264 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789197    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 08:39:35.803264 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789200    4473 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 08:39:35.803264 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789203    4473 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 08:39:35.803264 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789205    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 08:39:35.803264 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789208    4473 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 08:39:35.803264 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789210    4473 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 08:39:35.803264 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789213    4473 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 08:39:35.803264 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789215    4473 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 08:39:35.803264 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789217    4473 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 08:39:35.803264 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789220    4473 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 08:39:35.803264 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789223    4473 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 08:39:35.803264 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789225    4473 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 08:39:35.803795 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789229    4473 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 08:39:35.803795 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789231    4473 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 08:39:35.803795 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789234    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 08:39:35.803795 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789236    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 08:39:35.803795 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.789239    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 08:39:35.803795 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.795026    4473 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 08:39:35.803795 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.795046    4473 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 08:39:35.803795 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.795064    4473 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 08:39:35.803795 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.795069    4473 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 08:39:35.803795 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795133    4473 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 08:39:35.803795 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795139    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 08:39:35.803795 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795142    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 08:39:35.803795 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795145    4473 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 08:39:35.803795 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795148    4473 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 08:39:35.803795 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795151    4473 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 08:39:35.803795 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795153    4473 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 08:39:35.803795 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795157    4473 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 08:39:35.803795 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795160    4473 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 08:39:35.803795 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795163    4473 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 08:39:35.803795 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795165    4473 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 08:39:35.804312 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795168    4473 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 08:39:35.804312 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795171    4473 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 08:39:35.804312 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795173    4473 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 08:39:35.804312 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795176    4473 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 08:39:35.804312 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795178    4473 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 08:39:35.804312 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795181    4473 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 08:39:35.804312 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795184    4473 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 08:39:35.804312 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795187    4473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 08:39:35.804312 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795189    4473 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 08:39:35.804312 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795192    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 08:39:35.804312 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795194    4473 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 08:39:35.804312 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795197    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 08:39:35.804312 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795199    4473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 08:39:35.804312 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795202    4473 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 08:39:35.804312 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795205    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 08:39:35.804312 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795208    4473 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 08:39:35.804312 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795211    4473 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 08:39:35.804312 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795214    4473 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 08:39:35.804312 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795217    4473 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 08:39:35.804312 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795220    4473 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 08:39:35.804842 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795223    4473 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 08:39:35.804842 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795226    4473 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 08:39:35.804842 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795228    4473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 08:39:35.804842 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795231    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 08:39:35.804842 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795233    4473 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 08:39:35.804842 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795236    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 08:39:35.804842 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795239    4473 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 08:39:35.804842 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795241    4473 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 08:39:35.804842 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795244    4473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 08:39:35.804842 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795246    4473 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 08:39:35.804842 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795249    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 08:39:35.804842 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795251    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 08:39:35.804842 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795254    4473 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 08:39:35.804842 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795256    4473 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 08:39:35.804842 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795259    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 08:39:35.804842 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795261    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 08:39:35.804842 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795264    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 08:39:35.804842 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795266    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 08:39:35.804842 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795269    4473 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 08:39:35.805327 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795273    4473 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 08:39:35.805327 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795277    4473 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 08:39:35.805327 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795280    4473 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 08:39:35.805327 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795282    4473 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 08:39:35.805327 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795285    4473 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 08:39:35.805327 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795287    4473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 08:39:35.805327 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795290    4473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 08:39:35.805327 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795292    4473 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 08:39:35.805327 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795295    4473 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 08:39:35.805327 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795298    4473 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 08:39:35.805327 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795300    4473 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 08:39:35.805327 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795303    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 08:39:35.805327 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795305    4473 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 08:39:35.805327 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795308    4473 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 08:39:35.805327 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795310    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 08:39:35.805327 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795313    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 08:39:35.805327 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795315    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 08:39:35.805327 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795318    4473 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 08:39:35.805327 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795320    4473 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 08:39:35.805791 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795323    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 08:39:35.805791 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795326    4473 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 08:39:35.805791 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795329    4473 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 08:39:35.805791 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795331    4473 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 08:39:35.805791 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795334    4473 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 08:39:35.805791 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795336    4473 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 08:39:35.805791 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795339    4473 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 08:39:35.805791 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795341    4473 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 08:39:35.805791 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795344    4473 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 08:39:35.805791 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795346    4473 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 08:39:35.805791 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795349    4473 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 08:39:35.805791 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795352    4473 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 08:39:35.805791 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795355    4473 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 08:39:35.805791 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795357    4473 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 08:39:35.805791 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795360    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 08:39:35.805791 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795362    4473 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 08:39:35.805791 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795365    4473 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 08:39:35.805791 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795367    4473 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 08:39:35.805791 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795370    4473 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 08:39:35.805791 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795372    4473 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 08:39:35.806361 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795375    4473 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 08:39:35.806361 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795378    4473 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 08:39:35.806361 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795380    4473 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 08:39:35.806361 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795382    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 08:39:35.806361 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795385    4473 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 08:39:35.806361 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795388    4473 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 08:39:35.806361 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795390    4473 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 08:39:35.806361 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795393    4473 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 08:39:35.806361 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795395    4473 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 08:39:35.806361 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795398    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 08:39:35.806361 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795401    4473 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 08:39:35.806361 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795404    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 08:39:35.806361 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795406    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 08:39:35.806361 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795410    4473 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 08:39:35.806361 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795413    4473 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 08:39:35.806361 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795417    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 08:39:35.806361 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795419    4473 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 08:39:35.806772 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.795448    4473 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 08:39:35.806772 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795603    4473 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 08:39:35.806772 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795608    4473 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 08:39:35.806772 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795611    4473 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 08:39:35.806772 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795613    4473 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 08:39:35.806772 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795617    4473 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 08:39:35.806772 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795622    4473 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 08:39:35.806772 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795626    4473 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 08:39:35.806772 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795628    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 08:39:35.806772 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795632    4473 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 08:39:35.806772 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795634    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 08:39:35.806772 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795637    4473 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 08:39:35.806772 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795640    4473 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 08:39:35.806772 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795642    4473 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 08:39:35.806772 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795645    4473 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 08:39:35.807152 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795647    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 08:39:35.807152 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795650    4473 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 08:39:35.807152 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795652    4473 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 08:39:35.807152 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795655    4473 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 08:39:35.807152 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795657    4473 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 08:39:35.807152 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795660    4473 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 08:39:35.807152 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795662    4473 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 08:39:35.807152 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795664    4473 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 08:39:35.807152 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795667    4473 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 08:39:35.807152 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795669    4473 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 08:39:35.807152 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795672    4473 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 08:39:35.807152 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795674    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 08:39:35.807152 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795677    4473 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 08:39:35.807152 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795680    4473 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 08:39:35.807152 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795682    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 08:39:35.807152 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795685    4473 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 08:39:35.807152 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795687    4473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 08:39:35.807152 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795690    4473 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 08:39:35.807152 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795692    4473 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 08:39:35.807152 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795695    4473 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 08:39:35.807626 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795697    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 08:39:35.807626 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795700    4473 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 08:39:35.807626 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795702    4473 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 08:39:35.807626 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795704    4473 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 08:39:35.807626 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795707    4473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 08:39:35.807626 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795710    4473 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 08:39:35.807626 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795712    4473 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 08:39:35.807626 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795715    4473 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 08:39:35.807626 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795717    4473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 08:39:35.807626 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795720    4473 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 08:39:35.807626 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795722    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 08:39:35.807626 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795725    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 08:39:35.807626 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795727    4473 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 08:39:35.807626 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795729    4473 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 08:39:35.807626 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795733    4473 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 08:39:35.807626 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795737    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 08:39:35.807626 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795739    4473 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 08:39:35.807626 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795742    4473 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 08:39:35.807626 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795744    4473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 08:39:35.808101 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795747    4473 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 08:39:35.808101 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795749    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 08:39:35.808101 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795752    4473 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 08:39:35.808101 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795754    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 08:39:35.808101 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795757    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 08:39:35.808101 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795759    4473 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 08:39:35.808101 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795762    4473 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 08:39:35.808101 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795764    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 08:39:35.808101 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795766    4473 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 08:39:35.808101 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795769    4473 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 08:39:35.808101 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795771    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 08:39:35.808101 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795773    4473 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 08:39:35.808101 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795776    4473 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 08:39:35.808101 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795778    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 08:39:35.808101 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795780    4473 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 08:39:35.808101 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795783    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 08:39:35.808101 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795785    4473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 08:39:35.808101 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795788    4473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 08:39:35.808101 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795791    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 08:39:35.808101 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795793    4473 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 08:39:35.808579 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795796    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 08:39:35.808579 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795798    4473 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 08:39:35.808579 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795801    4473 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 08:39:35.808579 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795803    4473 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 08:39:35.808579 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795805    4473 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 08:39:35.808579 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795808    4473 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 08:39:35.808579 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795810    4473 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 08:39:35.808579 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795812    4473 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 08:39:35.808579 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795815    4473 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 08:39:35.808579 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795817    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 08:39:35.808579 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795820    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 08:39:35.808579 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795822    4473 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 08:39:35.808579 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795825    4473 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 08:39:35.808579 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795829    4473 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 08:39:35.808579 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795832    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 08:39:35.808579 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795834    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 08:39:35.808579 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795837    4473 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 08:39:35.808579 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795856    4473 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 08:39:35.808579 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795860    4473 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 08:39:35.808579 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795863    4473 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 08:39:35.809098 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795866    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 08:39:35.809098 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795869    4473 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 08:39:35.809098 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795871    4473 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 08:39:35.809098 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795874    4473 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 08:39:35.809098 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795876    4473 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 08:39:35.809098 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795880    4473 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 08:39:35.809098 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795882    4473 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 08:39:35.809098 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795885    4473 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 08:39:35.809098 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795887    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 08:39:35.809098 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795890    4473 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 08:39:35.809098 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795893    4473 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 08:39:35.809098 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795896    4473 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 08:39:35.809098 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795898    4473 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 08:39:35.809098 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.795921    4473 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 08:39:35.809098 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795972    4473 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 08:39:35.809098 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795977    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 08:39:35.809505 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795980    4473 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 08:39:35.809505 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795983    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 08:39:35.809505 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795986    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 08:39:35.809505 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795988    4473 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 08:39:35.809505 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795990    4473 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 08:39:35.809505 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795993    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 08:39:35.809505 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795995    4473 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 08:39:35.809505 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.795998    4473 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 08:39:35.809505 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796000    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 08:39:35.809505 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796003    4473 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 08:39:35.809505 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796006    4473 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 08:39:35.809505 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796009    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 08:39:35.809505 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796011    4473 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 08:39:35.809505 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796014    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 08:39:35.809505 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796017    4473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 08:39:35.809505 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796019    4473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 08:39:35.809505 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796021    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 08:39:35.809505 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796024    4473 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 08:39:35.809505 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796026    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 08:39:35.809505 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796029    4473 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 08:39:35.809994 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796031    4473 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 08:39:35.809994 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796033    4473 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 08:39:35.809994 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796037    4473 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 08:39:35.809994 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796040    4473 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 08:39:35.809994 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796042    4473 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 08:39:35.809994 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796044    4473 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 08:39:35.809994 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796047    4473 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 08:39:35.809994 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796049    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 08:39:35.809994 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796052    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 08:39:35.809994 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796055    4473 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 08:39:35.809994 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796057    4473 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 08:39:35.809994 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796060    4473 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 08:39:35.809994 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796063    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 08:39:35.809994 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796065    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 08:39:35.809994 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796067    4473 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 08:39:35.809994 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796070    4473 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 08:39:35.809994 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796073    4473 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 08:39:35.809994 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796075    4473 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 08:39:35.809994 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796077    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 08:39:35.809994 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796080    4473 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 08:39:35.810482 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796082    4473 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 08:39:35.810482 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796085    4473 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 08:39:35.810482 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796087    4473 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 08:39:35.810482 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796090    4473 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 08:39:35.810482 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796093    4473 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 08:39:35.810482 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796095    4473 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 08:39:35.810482 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796098    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 08:39:35.810482 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796101    4473 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 08:39:35.810482 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796103    4473 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 08:39:35.810482 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796105    4473 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 08:39:35.810482 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796108    4473 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 08:39:35.810482 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796110    4473 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 08:39:35.810482 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796113    4473 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 08:39:35.810482 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796115    4473 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 08:39:35.810482 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796119    4473 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 08:39:35.810482 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796123    4473 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 08:39:35.810482 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796127    4473 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 08:39:35.810482 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796130    4473 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 08:39:35.810482 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796133    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 08:39:35.811377 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796135    4473 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 08:39:35.811377 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796138    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 08:39:35.811377 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796141    4473 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 08:39:35.811377 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796143    4473 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 08:39:35.811377 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796146    4473 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 08:39:35.811377 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796148    4473 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 08:39:35.811377 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796151    4473 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 08:39:35.811377 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796153    4473 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 08:39:35.811377 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796156    4473 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 08:39:35.811377 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796158    4473 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 08:39:35.811377 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796161    4473 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 08:39:35.811377 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796163    4473 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 08:39:35.811377 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796166    4473 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 08:39:35.811377 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796168    4473 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 08:39:35.811377 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796171    4473 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 08:39:35.811377 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796173    4473 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 08:39:35.811377 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796176    4473 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 08:39:35.811377 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796179    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 08:39:35.811377 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796181    4473 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 08:39:35.811377 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796184    4473 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 08:39:35.811980 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796186    4473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 08:39:35.811980 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796189    4473 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 08:39:35.811980 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796191    4473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 08:39:35.811980 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796193    4473 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 08:39:35.811980 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796196    4473 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 08:39:35.811980 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796198    4473 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 08:39:35.811980 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796200    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 08:39:35.811980 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796204    4473 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 08:39:35.811980 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796206    4473 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 08:39:35.811980 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796209    4473 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 08:39:35.811980 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796211    4473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 08:39:35.811980 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796214    4473 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 08:39:35.811980 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796216    4473 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 08:39:35.811980 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796219    4473 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 08:39:35.811980 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796221    4473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 08:39:35.811980 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796225    4473 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 08:39:35.811980 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796227    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 08:39:35.811980 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796229    4473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 08:39:35.811980 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796232    4473 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 08:39:35.811980 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796234    4473 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 08:39:35.812496 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796239    4473 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 08:39:35.812496 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796243    4473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 08:39:35.812496 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796245    4473 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 08:39:35.812496 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796247    4473 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 08:39:35.812496 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:35.796250    4473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 08:39:35.812496 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.796326    4473 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 08:39:35.812496 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.797302    4473 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 08:39:35.812496 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.797361    4473 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 08:39:35.812496 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.797534    4473 server.go:1014] "Starting client certificate rotation"
Nov 03 08:39:35.812496 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.797724    4473 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 08:39:35.812496 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.797782    4473 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 08:33:55 +0000 UTC" deadline="2028-06-24 18:41:51.882048646 +0000 UTC"
Nov 03 08:39:35.812496 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.797859    4473 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14386h2m16.084192667s"
Nov 03 08:39:35.812496 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.798170    4473 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 08:39:35.812496 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.799290    4473 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 08:39:35.812496 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.800901    4473 log.go:25] "Validated CRI v1 runtime API"
Nov 03 08:39:35.812496 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.802449    4473 log.go:25] "Validated CRI v1 image API"
Nov 03 08:39:35.812886 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.803171    4473 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 08:39:35.812886 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.804741    4473 fs.go:135] Filesystem UUIDs: map[4e387ece-1ded-4257-a10a-c694e7b98d46:/dev/nvme0n1p3 669aaf52-edbc-4c40-9a6a-bb3102824100:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Nov 03 08:39:35.812973 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.804760    4473 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2839b31049e8b3e9fdc8ebada29c664da019834a925a4f9452314d958932725d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2839b31049e8b3e9fdc8ebada29c664da019834a925a4f9452314d958932725d/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2fb790962bf7280fc528b5620163aea464dd153e30db2c6bfb1081c97c98dd01/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2fb790962bf7280fc528b5620163aea464dd153e30db2c6bfb1081c97c98dd01/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3cacd773f4794a07bd9127962388165da3304c7adda2ab3b7548a9a22f49ac05/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3cacd773f4794a07bd9127962388165da3304c7adda2ab3b7548a9a22f49ac05/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3ddf1aaa0fa51521a2ebb29aada9c9536366809baf767c1d14faec6b34055f75/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3ddf1aaa0fa51521a2ebb29aada9c9536366809baf767c1d14faec6b34055f75/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/814fc1234beb8c798254b95bb56f77b9ee9abee1c9b30e42e7aa077c3cc3f0e5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/814fc1234beb8c798254b95bb56f77b9ee9abee1c9b30e42e7aa077c3cc3f0e5/userdata/shm major:0 minor:211 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/83db3cb481cbf0cce124c9d83fecae81c881a9b35d4f3a701e460b34d7e2124f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/83db3cb481cbf0cce124c9d83fecae81c881a9b35d4f3a701e460b34d7e2124f/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/977e7973a68f0a7bba883c7e4ec44a6f55a546c7f6a168b507f771f85a13c3ab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/977e7973a68f0a7bba883c7e4ec44a6f55a546c7f6a168b507f771f85a13c3ab/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b580705cc718f166d5784f6f78ecc317a4b988d9a0cae191724a15bd2df10c5b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b580705cc718f166d5784f6f78ecc317a4b988d9a0cae191724a15bd2df10c5b/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c020da724b255d14c6a1e90b1d4cfa4e3f76de617fcd0ceaca2564f8abcc2c22/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c020da724b255d14c6a1e90b1d4cfa4e3f76de617fcd0ceaca2564f8abcc2c22/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d3f5442cc25a0226fe87ab23a8535a38b81051a06246180dca6beacd8ab7d17e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d3f5442cc25a0226fe87ab23a8535a38b81051a06246180dca6beacd8ab7d17e/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/da7a63703f623ef740358ec23a7ea0a5619ff7c821233a8c0790d67dbfb5661f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/da7a63703f623ef740358ec23a7ea0a5619ff7c821233a8c0790d67dbfb5661f/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/de355649162da193e299f0a349b7f80afd241a4b5c7f8dfb7b38d5dfb7077678/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/de355649162da193e299f0a349b7f80afd241a4b5c7f8dfb7b38d5dfb7077678/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e6ed42962fc5432b2d8e7c837373b8999404b2c6b6361d2f782824a103e27830/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e6ed42962fc5432b2d8e7c837373b8999404b2c6b6361d2f782824a103e27830/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e7f26d2e538a5750f84d313224064ca989b465ae6d4252c818e516fdbdfa8310/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e7f26d2e538a5750f84d313224064ca989b465ae6d4252c818e516fdbdfa8310/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fe94beb078f134b2903a09f260a7c1330be776be4ae4520d84ffda96252c84ca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fe94beb078f134b2903a09f260a7c1330be776be4ae4520d84ffda96252c84ca/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a5d9c55-0799-4ae1-a902-dc06f813907c/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/0a5d9c55-0799-4ae1-a902-dc06f813907c/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a5d9c55-0799-4ae1-a902-dc06f813907c/volumes/kubernetes.io~projected/kube-api-access-s4vs4:{mountpoint:/var/lib/kubelet/pods/0a5d9c55-0799-4ae1-a902-dc06f813907c/volumes/kubernetes.io~projected/kube-api-access-s4vs4 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a5d9c55-0799-4ae1-a902-dc06f813907c/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/0a5d9c55-0799-4ae1-a902-dc06f813907c/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/105e26de-f1f1-4aae-99ef-e6a9491ccbe8/volumes/kubernetes.io~projected/kube-api-access-8bg9b:{mountpoint:/var/lib/kubelet/pods/105e26de-f1f1-4aae-99ef-e6a9491ccbe8/volumes/kubernetes.io~projected/kube-api-access-8bg9b major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/110651f3-9a70-43bf-888d-0813682bc891/volumes/kubernetes.io~projected/kube-api-access-hpfx2:{mountpoint:/var/lib/kubelet/pods/110651f3-9a70-43bf-888d-0813682bc891/volumes/kubernetes.io~projected/kube-api-access-hpfx2 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49fbaf82-5d99-48e1-90eb-005aec9f3d1b/volumes/kubernetes.io~projected/kube-api-access-swvhl:{mountpoint:/var/lib/kubelet/pods/49fbaf82-5d99-48e1-90eb-005aec9f3d1b/volumes/kubernetes.io~projected/kube-api-access-swvhl major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49fbaf82-5d99-48e1-90eb-005aec9f3d1b/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/49fbaf82-5d99-48e1-90eb-005aec9f3d1b/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b529371-cf9d-40e4-a2c5-55d69ab9a08d/volumes/kubernetes.io~projected/kube-api-access-sgfkn:{mountpoint:/var/lib/kubelet/pods/6b529371-cf9d-40e4-a2c5-55d69ab9a08d/volumes/kubernetes.io~projected/kube-api-access-sgfkn major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c26dfa4-e47e-43a1-94a2-816d22fb7f3a/volumes/kubernetes.io~projected/kube-api-access-99jxr:{mountpoint:/var/lib/kubelet/pods/7c26dfa4-e47e-43a1-94a2-816d22fb7f3a/volumes/kubernetes.io~projected/kube-api-access-99jxr major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e02a4de-6df2-4930-8360-4c5e09897472/volumes/kubernetes.io~projected/kube-api-access-jvsbx:{mountpoint:/var/lib/kubelet/pods/7e02a4de-6df2-4930-8360-4c5e09897472/volumes/kubernetes.io~projected/kube-api-access-jvsbx major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/801881d6-f217-4c80-a190-af953df3af04/volumes/kubernetes.io~projected/kube-api-access-njvj8:{mountpoint:/var/lib/kubelet/pods/801881d6-f217-4c80-a190-af953df3af04/volumes/kubernetes.io~projected/kube-api-access-njvj8 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/906c75b6-325c-4e89-8a10-e4e355884133/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/906c75b6-325c-4e89-8a10-e4e355884133/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9/volumes/kubernetes.io~projected/kube-api-access-7rxtf:{mountpoint:/var/lib/kubelet/pods/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9/volumes/kubernetes.io~projected/kube-api-access-7rxtf major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e8945a0-1bd9-4179-8996-a165bb1db1c4/volumes/kubernetes.io~projected/kube-api-access-7nz7q:{mountpoint:/var/lib/kubelet/pods/9e8945a0-1bd9-4179-8996-a165bb1db1c4/volumes/kubernetes.io~projected/kube-api-access-7nz7q major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e8945a0-1bd9-4179-8996-a165bb1db1c4/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/9e8945a0-1bd9-4179-8996-a165bb1db1c4/volumes/kubernetes.io~secret/metrics-tls major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1649ad9-921c-49d9-baa5-c51f44175b8c/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/d1649ad9-921c-49d9-baa5-c51f44175b8c/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1649ad9-921c-49d9-baa5-c51f44175b8c/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/d1649ad9-921c-49d9-baa5-c51f44175b8c/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1649ad9-921c-49d9-baa5-c51f44175b8c/volumes/kubernetes.io~projected/kube-api-access-xnnxt:{mountpoint:/var/lib/kubelet/pods/d1649ad9-921c-49d9-baa5-c51f44175b8c/volumes/kubernetes.io~projected/kube-api-access-xnnxt major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7329659-0fc1-4d97-b321-11e90bf7d13c/volumes/kubernetes.io~projected/kube-api-access-kv94t:{mountpoint:/var/lib/kubelet/pods/d7329659-0fc1-4d97-b321-11e90bf7d13c/volumes/kubernetes.io~projected/kube-api-access-kv94t major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7329659-0fc1-4d97-b321-11e90bf7d13c/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/d7329659-0fc1-4d97-b321-11e90bf7d13c/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1386760-3800-418f-bb1b-20e19880620b/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/e1386760-3800-418f-bb1b-20e19880620b/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6ca3dc3-231e-4a8d-a16c-7384c225110a/volumes/kubernetes.io~projected/kube-api-access-fcm5z:{mountpoint:/var/lib/kubelet/pods/f6ca3dc3-231e-4a8d-a16c-7384c225110a/volumes/kubernetes.io~projected/kube-api-access-fcm5z major:0 minor:208 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/a81e1b585653ad71a87b058623c98ef647acba7e3421db6fa06be1727fbdec15/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/a0c35a14de656528951ff502240bb90622fe952f5e491fb8d02894076d824091/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/f92e68cd16f04ce17d4e106cca35c89543375779b72abcf01806b1a72e458ce3/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/bab8eb75e955e6299f22c8f70c9e9e866da8be2508b22010adea963b231534f4/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/0bbc48b0e022e9191916e0e3aaa8c9acebdd99b69814c11b65828cd39e346cc2/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/a749cb9db76158c55d793f3fd78714bd5564da167b6ae36e1b70f6d5e16c69f8/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/9f016d55d5f859aaee35cb50d7c2f218560bce2be66c33e3d145e529cd41be07/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/7c1c372e472ec7ad528d0dcf13eeeb396f7d5ba19fe7c7b5f5a892b6add21f67/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/a496c5f39875540381a60553f586624caed9d101803c15b452bdbce063b4f57a/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/93da96912e8a3ac52839fee8fa73117613d36e8518dc5ce1eea08991fd62555f/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/d173aa23f70b58b8e866c08c1131807b1f20b9b50baa878c967ab6781db1a664/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/b3208a82a7185f8d88dbc3de1bdfbb75eefc96edf901bfa29f8cd3b590a5df33/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/eb2edafcfdcb174cf6728a04837c02d5708def3545ea4678d2b2c96c593689e2/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/39d50c68177ecf25dd962d213fea900b7bb71b0f6ed5eeaa16aeda468c10768b/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-158:{mountpoint:/var/lib/containers/storage/overlay/2b86045bbef20ddcde05d63dff4b13c0e4dd93687aa680dd5b2ac0380357c7a8/merged major:0 minor:158 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/d3c7da86a651047d736f285d75a157cca1bc73abae2def0567385fcb53649b93/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/7ada6e19a01bd0aaeae8b89c3d444618a506594cfab335bfb2a6eed9ffd0bb5d/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/d616ba7313c2fe906bbe0b36661d59f47d5db6c1d79ced4342c4abd3ffc53475/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/865cab8e7cc535b779e722b2deae061fbe45da8787d94392d9a92e6d421d0e22/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/3802c4a88a30acc580f127c6f3198e389f797a7be4752345e42185a57563b3af/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/1bb74c8ea9a704675ba8494b593c9f889304815a6c6b3cd4322e9b8d332c7d79/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/085d6e729f3e764044b72d0d7b49725536ae4685e49767f3e6f86e807a2e0ecd/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/878cb90f3d6cc74cf7296c4ea24f3890997a8d12c08bd9dd0cc431e2b4f04a2b/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/065e357cb66ac042c404b5771a54beb138f0642936aebd1639ec71dae52e2a68/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/0120cdb6159cc9fbe69116e2ec4ebac956f97155e30046850037dd457359d043/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/7012bdd8be2dc4554c82ad9d6c5984b570233cdc86c1e85f93016f725df7cf63/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/a408756db7788e9e8fc97dfaa1fe167dfcb0c0e52ed2dc1be737131b60b29514/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/c451c9347099111c3db56e158e7ac4ba7716134a0101a76eafc3d36e31de2755/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/19fc48087af5e219255c9357f4762eb328cf3c0a9f711bd0ac3224e55c74fa01/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/760f7c295cb6a68c57efdbefe3151924b24654d876860178f692f820872ac350/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/5e1f4b219fbe28b1d2d87fdcf45e2dd1ddf8568303a40ec428050e68493529fd/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/ea7390b8eaba9a850460134b4b266959a079c002e91339e2f5485e2c0cd3bb8f/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-71:{mountpoint:/var/lib/containers/storage/overlay/e5c6d0a847795b0862ebcce2b5a2b1436269e3f30318ec5a63005906081dbb20/merged major:0 minor:71 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/45d32a36a21eb934e34c596ccb9133975ae66273ebbcbd195a1af6678241dd97/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/3a5e7bb29ac2d2d50bf0984265556af485fe91231043cc50aed438ea7169aa33/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/8af454d3b918b79ce9abe3f2bd9cc5909c98ae669b21651b5cf2b685b50064d7/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/79313a94178325a9dc75de940b140ba2f20c7b3927132b9fa2ec7a6a7e5390be/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/782eb0b1b4e821cc16846d0199fdb6d3176af4e5fb82f5d7e3c27808a5b9761a/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-93:{mountpoint:/var/lib/containers/storage/overlay/3633c101f4124279db87a4fe0e5e9f96c5ff819d40f5e95d51cdeed238c52701/merged major:0 minor:93 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/477ea23ed15042412b42f7dff5ae08d6a572f2902e2a8a3775563de0a8a5f830/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 08:39:35.813165 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.812877    4473 manager.go:217] Machine: {Timestamp:2026-11-03 08:39:35.812387266 +0000 UTC m=+0.081450708 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102060 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec297d172f32aa054a5dac938116ab5a SystemUUID:ec297d17-2f32-aa05-4a5d-ac938116ab5a BootID:0992e8ce-4269-4c76-bfa9-9413680324e8 Filesystems:[{Device:/run/containers/storage/overlay-containers/e7f26d2e538a5750f84d313224064ca989b465ae6d4252c818e516fdbdfa8310/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2839b31049e8b3e9fdc8ebada29c664da019834a925a4f9452314d958932725d/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/7c26dfa4-e47e-43a1-94a2-816d22fb7f3a/volumes/kubernetes.io~projected/kube-api-access-99jxr DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/110651f3-9a70-43bf-888d-0813682bc891/volumes/kubernetes.io~projected/kube-api-access-hpfx2 DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9/volumes/kubernetes.io~projected/kube-api-access-7rxtf DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-71 DeviceMajor:0 DeviceMinor:71 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/0a5d9c55-0799-4ae1-a902-dc06f813907c/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e6ed42962fc5432b2d8e7c837373b8999404b2c6b6361d2f782824a103e27830/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d1649ad9-921c-49d9-baa5-c51f44175b8c/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d7329659-0fc1-4d97-b321-11e90bf7d13c/volumes/kubernetes.io~projected/kube-api-access-kv94t DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/49fbaf82-5d99-48e1-90eb-005aec9f3d1b/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:223 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-93 DeviceMajor:0 DeviceMinor:93 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/977e7973a68f0a7bba883c7e4ec44a6f55a546c7f6a168b507f771f85a13c3ab/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b580705cc718f166d5784f6f78ecc317a4b988d9a0cae191724a15bd2df10c5b/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9e8945a0-1bd9-4179-8996-a165bb1db1c4/volumes/kubernetes.io~projected/kube-api-access-7nz7q DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/906c75b6-325c-4e89-8a10-e4e355884133/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:207 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fe94beb078f134b2903a09f260a7c1330be776be4ae4520d84ffda96252c84ca/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/de355649162da193e299f0a349b7f80afd241a4b5c7f8dfb7b38d5dfb7077678/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/814fc1234beb8c798254b95bb56f77b9ee9abee1c9b30e42e7aa077c3cc3f0e5/userdata/shm DeviceMajor:0 DeviceMinor:211 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3ddf1aaa0fa51521a2ebb29aada9c9536366809baf767c1d14faec6b34055f75/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c020da724b255d14c6a1e90b1d4cfa4e3f76de617fcd0ceaca2564f8abcc2c22/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-158 DeviceMajor:0 DeviceMinor:158 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e1386760-3800-418f-bb1b-20e19880620b/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0a5d9c55-0799-4ae1-a902-dc06f813907c/volumes/kubernetes.io~projected/kube-api-access-s4vs4 DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/105e26de-f1f1-4aae-99ef-e6a9491ccbe8/volumes/kubernetes.io~projected/kube-api-access-8bg9b DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6b529371-cf9d-40e4-a2c5-55d69ab9a08d/volumes/kubernetes.io~projected/kube-api-access-sgfkn DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/da7a63703f623ef740358ec23a7ea0a5619ff7c821233a8c0790d67dbfb5661f/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0a5d9c55-0799-4ae1-a902-dc06f813907c/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/7e02a4de-6df2-4930-8360-4c5e09897472/volumes/kubernetes.io~projected/kube-api-access-jvsbx DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2fb790962bf7280fc528b5620163aea464dd153e30db2c6bfb1081c97c98dd01/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9e8945a0-1bd9-4179-8996-a165bb1db1c4/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:224 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d1649ad9-921c-49d9-baa5-c51f44175b8c/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/801881d6-f217-4c80-a190-af953df3af04/volumes/kubernetes.io~projected/kube-api-access-njvj8 DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d3f5442cc25a0226fe87ab23a8535a38b81051a06246180dca6beacd8ab7d17e/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/83db3cb481cbf0cce124c9d83fecae81c881a9b35d4f3a701e460b34d7e2124f/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/49fbaf82-5d99-48e1-90eb-005aec9f3d1b/volumes/kubernetes.io~projected/kube-api-access-swvhl DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f6ca3dc3-231e-4a8d-a16c-7384c225110a/volumes/kubernetes.io~projected/kube-api-access-fcm5z DeviceMajor:0 DeviceMinor:208 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d1649ad9-921c-49d9-baa5-c51f44175b8c/volumes/kubernetes.io~projected/kube-api-access-xnnxt DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3cacd773f4794a07bd9127962388165da3304c7adda2ab3b7548a9a22f49ac05/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d7329659-0fc1-4d97-b321-11e90bf7d13c/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:2839b31049e8b3e MacAddress:d6:90:a0:f9:e3:7a Speed:10000 Mtu:8901} {Name:2fb790962bf7280 MacAddress:52:73:5a:be:ee:c4 Speed:10000 Mtu:8901} {Name:814fc1234beb8c7 MacAddress:0e:9b:8c:62:3b:8e Speed:10000 Mtu:8901} {Name:83db3cb481cbf0c MacAddress:1e:70:3f:64:45:cc Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:12:f8:92:65:b1 Speed:0 Mtu:9001} {Name:br-int MacAddress:7a:b5:40:db:4e:82 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:12:f8:92:65:b1 Speed:0 Mtu:9001} {Name:fe94beb078f134b MacAddress:8e:20:92:27:0e:90 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:3a:e8:e4:68:f5:86 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ea:43:23:91:bb:87 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 08:39:35.813288 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.813169    4473 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 08:39:35.813288 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.813228    4473 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 08:39:35.813524 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.813495    4473 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 08:39:35.813651 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.813526    4473 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-73.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 08:39:35.813691 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.813653    4473 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 08:39:35.813691 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.813661    4473 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 08:39:35.813691 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.813671    4473 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 08:39:35.813691 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.813691    4473 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 08:39:35.813793 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.813789    4473 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 08:39:35.814092 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.813979    4473 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 08:39:35.814092 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.814090    4473 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 08:39:35.814187 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.814126    4473 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 08:39:35.814187 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.814139    4473 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 08:39:35.814187 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.814150    4473 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 08:39:35.814187 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.814159    4473 kubelet.go:405] "Adding apiserver pod source"
Nov 03 08:39:35.814187 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.814166    4473 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 08:39:35.814905 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.814891    4473 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 08:39:35.814957 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.814908    4473 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 08:39:35.815491 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.815468    4473 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 08:39:35.824895 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.824879    4473 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 08:39:35.824895 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.824893    4473 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 08:39:35.825032 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.825020    4473 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 08:39:35.825219 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.825207    4473 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 08:39:35.825249 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.825238    4473 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 08:39:35.825341 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.825330    4473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 08:39:35.825373 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.825351    4473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 08:39:35.825373 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.825365    4473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 08:39:35.825427 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.825374    4473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 08:39:35.825427 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.825382    4473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 08:39:35.825427 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.825388    4473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 08:39:35.825427 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.825393    4473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 08:39:35.825427 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.825399    4473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 08:39:35.825427 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.825406    4473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 08:39:35.825427 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.825411    4473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 08:39:35.825631 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.825433    4473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 08:39:35.825631 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.825443    4473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 08:39:35.825631 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.825470    4473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 08:39:35.825631 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.825478    4473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 08:39:35.828797 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.828784    4473 server.go:1263] "Started kubelet"
Nov 03 08:39:35.828916 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.828849    4473 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 08:39:35.828996 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.828899    4473 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 08:39:35.828996 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.828987    4473 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 08:39:35.830973 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.829355    4473 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 08:39:35.830973 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.830051    4473 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 08:39:35.829373 ip-10-0-139-73 systemd[1]: Started Kubernetes Kubelet.
Nov 03 08:39:35.831882 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.831866    4473 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 08:39:35.831975 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.831886    4473 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 08:39:35.831975 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:39:35.831961    4473 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-73.us-west-1.compute.internal\" not found"
Nov 03 08:39:35.832070 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.831999    4473 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 08:39:35.832070 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.832008    4473 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 08:39:35.832070 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.832011    4473 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 08:33:57 +0000 UTC" deadline="2028-04-07 09:19:14.49375036 +0000 UTC"
Nov 03 08:39:35.832070 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.832034    4473 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12504h39m38.661720121s"
Nov 03 08:39:35.832540 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.832510    4473 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.833717    4473 factory.go:55] Registering systemd factory
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.833752    4473 factory.go:223] Registration of the systemd container factory successfully
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.834895    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a5d9c55-0799-4ae1-a902-dc06f813907c" volumeName="kubernetes.io/configmap/0a5d9c55-0799-4ae1-a902-dc06f813907c-ovnkube-config" seLinuxMountContext=""
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835079    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="110651f3-9a70-43bf-888d-0813682bc891" volumeName="kubernetes.io/projected/110651f3-9a70-43bf-888d-0813682bc891-kube-api-access-hpfx2" seLinuxMountContext=""
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835095    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e02a4de-6df2-4930-8360-4c5e09897472" volumeName="kubernetes.io/projected/7e02a4de-6df2-4930-8360-4c5e09897472-kube-api-access-jvsbx" seLinuxMountContext=""
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835108    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="801881d6-f217-4c80-a190-af953df3af04" volumeName="kubernetes.io/empty-dir/801881d6-f217-4c80-a190-af953df3af04-tmp-dir" seLinuxMountContext=""
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835126    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="906c75b6-325c-4e89-8a10-e4e355884133" volumeName="kubernetes.io/secret/906c75b6-325c-4e89-8a10-e4e355884133-original-pull-secret" seLinuxMountContext=""
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835220    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e8945a0-1bd9-4179-8996-a165bb1db1c4" volumeName="kubernetes.io/projected/9e8945a0-1bd9-4179-8996-a165bb1db1c4-kube-api-access-7nz7q" seLinuxMountContext=""
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835243    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1649ad9-921c-49d9-baa5-c51f44175b8c" volumeName="kubernetes.io/projected/d1649ad9-921c-49d9-baa5-c51f44175b8c-kube-api-access-xnnxt" seLinuxMountContext=""
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835278    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1386760-3800-418f-bb1b-20e19880620b" volumeName="kubernetes.io/secret/e1386760-3800-418f-bb1b-20e19880620b-agent-certs" seLinuxMountContext=""
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835296    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a5d9c55-0799-4ae1-a902-dc06f813907c" volumeName="kubernetes.io/configmap/0a5d9c55-0799-4ae1-a902-dc06f813907c-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835313    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a5d9c55-0799-4ae1-a902-dc06f813907c" volumeName="kubernetes.io/configmap/0a5d9c55-0799-4ae1-a902-dc06f813907c-env-overrides" seLinuxMountContext=""
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835342    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a5d9c55-0799-4ae1-a902-dc06f813907c" volumeName="kubernetes.io/projected/0a5d9c55-0799-4ae1-a902-dc06f813907c-kube-api-access-s4vs4" seLinuxMountContext=""
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835357    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a5d9c55-0799-4ae1-a902-dc06f813907c" volumeName="kubernetes.io/secret/0a5d9c55-0799-4ae1-a902-dc06f813907c-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835369    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49fbaf82-5d99-48e1-90eb-005aec9f3d1b" volumeName="kubernetes.io/configmap/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835379    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49fbaf82-5d99-48e1-90eb-005aec9f3d1b" volumeName="kubernetes.io/empty-dir/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-data-volume" seLinuxMountContext=""
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835402    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b529371-cf9d-40e4-a2c5-55d69ab9a08d" volumeName="kubernetes.io/projected/6b529371-cf9d-40e4-a2c5-55d69ab9a08d-kube-api-access-sgfkn" seLinuxMountContext=""
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835418    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9" volumeName="kubernetes.io/configmap/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835429    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9" volumeName="kubernetes.io/projected/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-kube-api-access-7rxtf" seLinuxMountContext=""
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835440    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6ca3dc3-231e-4a8d-a16c-7384c225110a" volumeName="kubernetes.io/projected/f6ca3dc3-231e-4a8d-a16c-7384c225110a-kube-api-access-fcm5z" seLinuxMountContext=""
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835455    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="105e26de-f1f1-4aae-99ef-e6a9491ccbe8" volumeName="kubernetes.io/configmap/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-cni-binary-copy" seLinuxMountContext=""
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835465    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="105e26de-f1f1-4aae-99ef-e6a9491ccbe8" volumeName="kubernetes.io/configmap/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-multus-daemon-config" seLinuxMountContext=""
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835508    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="105e26de-f1f1-4aae-99ef-e6a9491ccbe8" volumeName="kubernetes.io/projected/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-kube-api-access-8bg9b" seLinuxMountContext=""
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835522    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="110651f3-9a70-43bf-888d-0813682bc891" volumeName="kubernetes.io/configmap/110651f3-9a70-43bf-888d-0813682bc891-serviceca" seLinuxMountContext=""
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835534    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49fbaf82-5d99-48e1-90eb-005aec9f3d1b" volumeName="kubernetes.io/projected/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-kube-api-access-swvhl" seLinuxMountContext=""
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835551    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="801881d6-f217-4c80-a190-af953df3af04" volumeName="kubernetes.io/projected/801881d6-f217-4c80-a190-af953df3af04-kube-api-access-njvj8" seLinuxMountContext=""
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835662    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e8945a0-1bd9-4179-8996-a165bb1db1c4" volumeName="kubernetes.io/configmap/9e8945a0-1bd9-4179-8996-a165bb1db1c4-config-volume" seLinuxMountContext=""
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835677    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1649ad9-921c-49d9-baa5-c51f44175b8c" volumeName="kubernetes.io/empty-dir/d1649ad9-921c-49d9-baa5-c51f44175b8c-tmp" seLinuxMountContext=""
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835688    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49fbaf82-5d99-48e1-90eb-005aec9f3d1b" volumeName="kubernetes.io/secret/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835699    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c26dfa4-e47e-43a1-94a2-816d22fb7f3a" volumeName="kubernetes.io/projected/7c26dfa4-e47e-43a1-94a2-816d22fb7f3a-kube-api-access-99jxr" seLinuxMountContext=""
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835722    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e8945a0-1bd9-4179-8996-a165bb1db1c4" volumeName="kubernetes.io/secret/9e8945a0-1bd9-4179-8996-a165bb1db1c4-metrics-tls" seLinuxMountContext=""
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835738    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7329659-0fc1-4d97-b321-11e90bf7d13c" volumeName="kubernetes.io/projected/d7329659-0fc1-4d97-b321-11e90bf7d13c-kube-api-access-kv94t" seLinuxMountContext=""
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835749    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7329659-0fc1-4d97-b321-11e90bf7d13c" volumeName="kubernetes.io/secret/d7329659-0fc1-4d97-b321-11e90bf7d13c-cert" seLinuxMountContext=""
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835759    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1386760-3800-418f-bb1b-20e19880620b" volumeName="kubernetes.io/configmap/e1386760-3800-418f-bb1b-20e19880620b-konnectivity-ca" seLinuxMountContext=""
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835774    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9" volumeName="kubernetes.io/configmap/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835785    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1649ad9-921c-49d9-baa5-c51f44175b8c" volumeName="kubernetes.io/empty-dir/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-tuned" seLinuxMountContext=""
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835800    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b529371-cf9d-40e4-a2c5-55d69ab9a08d" volumeName="kubernetes.io/configmap/6b529371-cf9d-40e4-a2c5-55d69ab9a08d-iptables-alerter-script" seLinuxMountContext=""
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835811    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9" volumeName="kubernetes.io/configmap/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-cni-binary-copy" seLinuxMountContext=""
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835823    4473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e8945a0-1bd9-4179-8996-a165bb1db1c4" volumeName="kubernetes.io/empty-dir/9e8945a0-1bd9-4179-8996-a165bb1db1c4-tmp-dir" seLinuxMountContext=""
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835836    4473 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835842    4473 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835938    4473 factory.go:153] Registering CRI-O factory
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.835975    4473 factory.go:223] Registration of the crio container factory successfully
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.836027    4473 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.836048    4473 factory.go:103] Registering Raw factory
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.836064    4473 manager.go:1199] Started watching for new ooms in manager
Nov 03 08:39:35.840295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.836594    4473 manager.go:319] Starting recovery of all containers
Nov 03 08:39:35.851373 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.851289    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 08:39:35.852140 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.852121    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 08:39:35.860762 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.860729    4473 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 08:39:35.862106 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.862085    4473 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 08:39:35.862187 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.862109    4473 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 08:39:35.862187 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.862130    4473 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 08:39:35.862187 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:39:35.862164    4473 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 08:39:35.869333 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.869313    4473 generic.go:356] "Generic (PLEG): container finished" podID="bda1773f2a5825c17c394ccc2e58c1ce" containerID="89c2c10538271e6aa71010aab9533709f01fc8653ca6ff409f44c310fa997afb" exitCode=0
Nov 03 08:39:35.873972 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.873930    4473 generic.go:356] "Generic (PLEG): container finished" podID="968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9" containerID="7e09b678185b447d6f4012ef095b0b5e5ba6e5fac2b3697eff203580a4eed654" exitCode=0
Nov 03 08:39:35.874043 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.873973    4473 generic.go:356] "Generic (PLEG): container finished" podID="968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9" containerID="e4382c2a8bdc72875523687fcd7fc301f8420d7fa5157f0c93fdda3185e11a5a" exitCode=0
Nov 03 08:39:35.874043 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.873985    4473 generic.go:356] "Generic (PLEG): container finished" podID="968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9" containerID="4a82dfa1b9438238ba347e874e04df130aebbbd52027093a07fb72f272483a27" exitCode=0
Nov 03 08:39:35.874043 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.873991    4473 generic.go:356] "Generic (PLEG): container finished" podID="968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9" containerID="d905d4d84b96574c3372a719706b6cac6f68e70410fe1f9828519895412776ee" exitCode=0
Nov 03 08:39:35.874043 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.873996    4473 generic.go:356] "Generic (PLEG): container finished" podID="968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9" containerID="090e8940649456277cd57142c0c6042a804ea921e5f62d3a371c971fbacd0c6f" exitCode=0
Nov 03 08:39:35.874043 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.874002    4473 generic.go:356] "Generic (PLEG): container finished" podID="968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9" containerID="1542ec1a8594d66dea43650dc095dad2d387f3ffe951d7a566916aba30b855db" exitCode=0
Nov 03 08:39:35.894173 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.894152    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 08:39:35.894513 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.894483    4473 generic.go:356] "Generic (PLEG): container finished" podID="0a5d9c55-0799-4ae1-a902-dc06f813907c" containerID="ebce2c73bb3f0fe5d0fcbb1344c342a30e2bc04e87b103a31f437aa19b12a2af" exitCode=1
Nov 03 08:39:35.901423 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.901406    4473 manager.go:324] Recovery completed
Nov 03 08:39:35.915438 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.915422    4473 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 08:39:35.916404 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.916389    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 08:39:35.916464 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.916419    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 08:39:35.916464 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.916434    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 08:39:35.917659 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.917642    4473 cpu_manager.go:226] "Starting" policy="none"
Nov 03 08:39:35.917707 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.917659    4473 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 08:39:35.917707 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.917676    4473 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 08:39:35.917769 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.917650    4473 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 08:39:35.917841 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.917824    4473 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 08:39:35.917876 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.917845    4473 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 08:39:35.917908 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.917880    4473 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 08:39:35.917908 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.917886    4473 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 08:39:35.917908 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.917892    4473 policy_none.go:50] "Start"
Nov 03 08:39:35.917908 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.917900    4473 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 08:39:35.918016 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.917909    4473 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 08:39:35.918433 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.918421    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 08:39:35.918496 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.918446    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 08:39:35.918496 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.918455    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 08:39:35.930359 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.930341    4473 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 08:39:35.930359 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.930357    4473 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 08:39:35.930451 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.930364    4473 policy_none.go:44] "Start"
Nov 03 08:39:35.932047 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:39:35.932032    4473 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-73.us-west-1.compute.internal\" not found"
Nov 03 08:39:35.934546 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.934516    4473 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 08:39:35.937798 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.937788    4473 manager.go:341] "Starting Device Plugin manager"
Nov 03 08:39:35.937843 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:39:35.937816    4473 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 08:39:35.937843 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.937828    4473 server.go:87] "Starting device plugin registration server"
Nov 03 08:39:35.938184 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.938173    4473 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 08:39:35.938214 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.938187    4473 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 08:39:35.938296 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.938274    4473 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 08:39:35.938430 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.938375    4473 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 08:39:35.938430 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.938412    4473 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 08:39:35.938430 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.938419    4473 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 08:39:35.938601 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.938472    4473 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T08:39:35.93840212Z","UUID":"0022bf2e-db50-4f4b-a134-751908c1c8c0","Handler":null,"Name":"","Endpoint":""}
Nov 03 08:39:35.939581 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:39:35.939565    4473 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 08:39:35.939672 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:39:35.939601    4473 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-73.us-west-1.compute.internal\" not found"
Nov 03 08:39:35.940011 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.939997    4473 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 08:39:35.940089 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.940017    4473 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 08:39:35.962906 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.962870    4473 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-73.us-west-1.compute.internal"]
Nov 03 08:39:35.963015 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.963003    4473 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 08:39:35.965717 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.963922    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 08:39:35.965717 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.964081    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 08:39:35.965717 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.964123    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 08:39:35.965717 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.964670    4473 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 08:39:35.965717 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.965046    4473 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 08:39:35.966589 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.966569    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 08:39:35.966693 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.966602    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 08:39:35.966693 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.966600    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 08:39:35.966693 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.966654    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 08:39:35.966693 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.966617    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 08:39:35.966693 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.966667    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 08:39:35.966878 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:39:35.966853    4473 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-73.us-west-1.compute.internal\" not found" node="ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:39:35.966955 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.966915    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal" event={"ID":"bda1773f2a5825c17c394ccc2e58c1ce","Type":"ContainerStarted","Data":"99cb3009262cc0a9abb3e55da11a02674c59bb9911766c21ed86a962bc5927b2"}
Nov 03 08:39:35.966988 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.966974    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal" event={"ID":"bda1773f2a5825c17c394ccc2e58c1ce","Type":"ContainerDied","Data":"89c2c10538271e6aa71010aab9533709f01fc8653ca6ff409f44c310fa997afb"}
Nov 03 08:39:35.967022 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.966990    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal" event={"ID":"bda1773f2a5825c17c394ccc2e58c1ce","Type":"ContainerStarted","Data":"3cacd773f4794a07bd9127962388165da3304c7adda2ab3b7548a9a22f49ac05"}
Nov 03 08:39:35.967054 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.967026    4473 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 08:39:35.967054 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.967036    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-73.us-west-1.compute.internal" event={"ID":"0f8947715126d508f609643819d8b2a7","Type":"ContainerStarted","Data":"5a3a3ea589cb9f3b3144012d6d55ea14cb7924dd6ebaae7b63351e1c57225b58"}
Nov 03 08:39:35.967054 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.967050    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-73.us-west-1.compute.internal" event={"ID":"0f8947715126d508f609643819d8b2a7","Type":"ContainerStarted","Data":"b580705cc718f166d5784f6f78ecc317a4b988d9a0cae191724a15bd2df10c5b"}
Nov 03 08:39:35.967680 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.967665    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 08:39:35.967729 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.967691    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 08:39:35.967729 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:35.967701    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 08:39:35.967826 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:39:35.967813    4473 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-73.us-west-1.compute.internal\" not found" node="ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:39:36.025831 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.025812    4473 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 08:39:36.027625 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.027610    4473 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 08:39:36.027722 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.027706    4473 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 08:39:36.033843 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.033830    4473 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:39:36.039004 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.038979    4473 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 08:39:36.039843 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.039824    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Nov 03 08:39:36.039920 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.039861    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 08:39:36.039920 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.039878    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeHasSufficientPID"
Nov 03 08:39:36.039920 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.039906    4473 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:39:36.078239 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.078219    4473 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 08:39:36.078373 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.078361    4473 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 08:39:36.125482 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.125462    4473 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 08:39:36.125530 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:39:36.125499    4473 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:39:36.125530 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.125510    4473 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:39:36.126567 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.126548    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bda1773f2a5825c17c394ccc2e58c1ce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal\" (UID: \"bda1773f2a5825c17c394ccc2e58c1ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:39:36.126663 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.126580    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bda1773f2a5825c17c394ccc2e58c1ce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal\" (UID: \"bda1773f2a5825c17c394ccc2e58c1ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:39:36.126663 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.126600    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0f8947715126d508f609643819d8b2a7-config\") pod \"kube-apiserver-proxy-ip-10-0-139-73.us-west-1.compute.internal\" (UID: \"0f8947715126d508f609643819d8b2a7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:39:36.178843 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.178794    4473 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:39:36.178915 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.178845    4473 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:39:36.179313 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.179300    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeNotReady"
Nov 03 08:39:36.179368 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.179326    4473 setters.go:546] "Node became not ready" node="ip-10-0-139-73.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-03T08:39:36Z","lastTransitionTime":"2026-11-03T08:39:36Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Nov 03 08:39:36.184695 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.184679    4473 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 08:39:36.184741 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:39:36.184717    4473 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-73.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:39:36.190995 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.190978    4473 nodeinfomanager.go:471] existing CSINode "ip-10-0-139-73.us-west-1.compute.internal" is owned by different node (oldNodeID="d9fec28f-ba8d-4a6b-b2d9-e06ac756b9e5", newNodeID=""), cleaning up...
Nov 03 08:39:36.227105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.227083    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bda1773f2a5825c17c394ccc2e58c1ce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal\" (UID: \"bda1773f2a5825c17c394ccc2e58c1ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:39:36.227164 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.227103    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bda1773f2a5825c17c394ccc2e58c1ce-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal\" (UID: \"bda1773f2a5825c17c394ccc2e58c1ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:39:36.227164 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.227119    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bda1773f2a5825c17c394ccc2e58c1ce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal\" (UID: \"bda1773f2a5825c17c394ccc2e58c1ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:39:36.227164 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.227137    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0f8947715126d508f609643819d8b2a7-config\") pod \"kube-apiserver-proxy-ip-10-0-139-73.us-west-1.compute.internal\" (UID: \"0f8947715126d508f609643819d8b2a7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:39:36.227164 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.227155    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bda1773f2a5825c17c394ccc2e58c1ce-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal\" (UID: \"bda1773f2a5825c17c394ccc2e58c1ce\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:39:36.227281 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.227173    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0f8947715126d508f609643819d8b2a7-config\") pod \"kube-apiserver-proxy-ip-10-0-139-73.us-west-1.compute.internal\" (UID: \"0f8947715126d508f609643819d8b2a7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:39:36.447538 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.447479    4473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-73.us-west-1.compute.internal" event="NodeReady"
Nov 03 08:39:36.447538 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.447543    4473 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 08:39:36.814909 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.814842    4473 apiserver.go:52] "Watching apiserver"
Nov 03 08:39:36.897301 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.897273    4473 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:39:36.897428 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.897369    4473 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:39:36.922046 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.921939    4473 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 08:39:36.923583 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.923556    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-clx84","openshift-insights/insights-runtime-extractor-tft2z","openshift-cluster-node-tuning-operator/tuned-jpll8","kube-system/konnectivity-agent-bq5p4","openshift-multus/multus-additional-cni-plugins-wsrmr","openshift-dns/node-resolver-hgbb5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal","openshift-network-operator/iptables-alerter-689v8","openshift-dns/dns-default-gvbpd","openshift-multus/multus-kzcqg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs","kube-system/kube-apiserver-proxy-ip-10-0-139-73.us-west-1.compute.internal","openshift-ingress-canary/ingress-canary-74dwn","openshift-ovn-kubernetes/ovnkube-node-bt4zt","openshift-multus/network-metrics-daemon-pflts","openshift-image-registry/node-ca-5drgw","kube-system/global-pull-secret-syncer-x9jpw"]
Nov 03 08:39:36.926621 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.926603    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pflts"
Nov 03 08:39:36.933403 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.933387    4473 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 08:39:36.956813 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.956796    4473 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 08:39:36.956868 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:39:36.956833    4473 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-73.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:39:36.958344 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.958330    4473 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 08:39:36.958387 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:39:36.958364    4473 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-73.us-west-1.compute.internal"
Nov 03 08:39:36.979653 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.979636    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jv5wj\""
Nov 03 08:39:36.980708 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.980667    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jnk2x\""
Nov 03 08:39:36.980824 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.980744    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8lmql\""
Nov 03 08:39:36.980824 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.980786    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 08:39:36.980824 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.980803    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 08:39:36.981060 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.980895    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 08:39:36.981060 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.980901    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hvj2z\""
Nov 03 08:39:36.981060 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.980897    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 08:39:36.981060 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.980910    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tps59\""
Nov 03 08:39:36.981060 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.980910    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sntnv\""
Nov 03 08:39:36.981060 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.980930    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 08:39:36.981060 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.980916    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 08:39:36.981060 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.980910    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jjcxd\""
Nov 03 08:39:36.981060 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.980989    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rnxpp\""
Nov 03 08:39:36.981060 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.981024    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 08:39:36.981393 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.981379    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 08:39:36.981569 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.981557    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 08:39:36.981885 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.981860    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 08:39:36.982011 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.981905    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 08:39:36.982011 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.982003    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 08:39:36.982193 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.982180    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 08:39:36.982506 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.982486    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 08:39:36.982506 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.982502    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 08:39:36.982668 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.982588    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 08:39:36.982668 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.982595    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 08:39:36.982668 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.982591    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 08:39:36.982668 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.982591    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 08:39:36.982668 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.982605    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 08:39:36.982668 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.982655    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 08:39:36.982991 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.982668    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 08:39:36.982991 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.982739    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 08:39:36.982991 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.982824    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cwzpz\""
Nov 03 08:39:36.982991 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.982881    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 08:39:36.982991 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.982939    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v9lhl\""
Nov 03 08:39:36.982991 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.982988    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zgn57\""
Nov 03 08:39:36.983268 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.983005    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 08:39:36.983268 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.983078    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 08:39:36.983268 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.983095    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 08:39:36.983268 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.983155    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 08:39:36.983268 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.983220    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-htlrf\""
Nov 03 08:39:36.983725 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.983709    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 08:39:36.983814 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.983801    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 08:39:36.983895 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.983883    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 08:39:36.983996 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.983985    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 08:39:36.984143 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.984131    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 08:39:36.989415 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.989399    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 08:39:36.989531 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.989515    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sxt9c\""
Nov 03 08:39:36.989576 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:36.989524    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 08:39:37.033282 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.033054    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-modprobe-d\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:39:37.033282 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.033109    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-kubernetes\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:39:37.033282 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.033138    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-host-var-lib-cni-multus\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.033282 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.033171    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-cnibin\") pod \"multus-additional-cni-plugins-wsrmr\" (UID: \"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9\") " pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:39:37.033282 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.033204    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7e02a4de-6df2-4930-8360-4c5e09897472-etc-selinux\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:39:37.033282 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.033242    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njvj8\" (UniqueName: \"kubernetes.io/projected/801881d6-f217-4c80-a190-af953df3af04-kube-api-access-njvj8\") pod \"node-resolver-hgbb5\" (UID: \"801881d6-f217-4c80-a190-af953df3af04\") " pod="openshift-dns/node-resolver-hgbb5"
Nov 03 08:39:37.033282 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.033273    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-systemd\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.033326    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-host\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.033358    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-cni-binary-copy\") pod \"multus-additional-cni-plugins-wsrmr\" (UID: \"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9\") " pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.033390    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7e02a4de-6df2-4930-8360-4c5e09897472-device-dir\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.033421    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgfkn\" (UniqueName: \"kubernetes.io/projected/6b529371-cf9d-40e4-a2c5-55d69ab9a08d-kube-api-access-sgfkn\") pod \"iptables-alerter-689v8\" (UID: \"6b529371-cf9d-40e4-a2c5-55d69ab9a08d\") " pod="openshift-network-operator/iptables-alerter-689v8"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.033451    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/110651f3-9a70-43bf-888d-0813682bc891-serviceca\") pod \"node-ca-5drgw\" (UID: \"110651f3-9a70-43bf-888d-0813682bc891\") " pod="openshift-image-registry/node-ca-5drgw"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.033480    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-slash\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.033527    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0a5d9c55-0799-4ae1-a902-dc06f813907c-env-overrides\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.033561    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9e8945a0-1bd9-4179-8996-a165bb1db1c4-config-volume\") pod \"dns-default-gvbpd\" (UID: \"9e8945a0-1bd9-4179-8996-a165bb1db1c4\") " pod="openshift-dns/dns-default-gvbpd"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.033595    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rxtf\" (UniqueName: \"kubernetes.io/projected/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-kube-api-access-7rxtf\") pod \"multus-additional-cni-plugins-wsrmr\" (UID: \"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9\") " pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.033624    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/801881d6-f217-4c80-a190-af953df3af04-hosts-file\") pod \"node-resolver-hgbb5\" (UID: \"801881d6-f217-4c80-a190-af953df3af04\") " pod="openshift-dns/node-resolver-hgbb5"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.033654    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-node-log\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.033689    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.033716    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e8945a0-1bd9-4179-8996-a165bb1db1c4-tmp-dir\") pod \"dns-default-gvbpd\" (UID: \"9e8945a0-1bd9-4179-8996-a165bb1db1c4\") " pod="openshift-dns/dns-default-gvbpd"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.033747    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv94t\" (UniqueName: \"kubernetes.io/projected/d7329659-0fc1-4d97-b321-11e90bf7d13c-kube-api-access-kv94t\") pod \"ingress-canary-74dwn\" (UID: \"d7329659-0fc1-4d97-b321-11e90bf7d13c\") " pod="openshift-ingress-canary/ingress-canary-74dwn"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.033778    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-crio-socket\") pod \"insights-runtime-extractor-tft2z\" (UID: \"49fbaf82-5d99-48e1-90eb-005aec9f3d1b\") " pod="openshift-insights/insights-runtime-extractor-tft2z"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.033807    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-host-var-lib-kubelet\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.033831    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-multus-cni-dir\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.033861    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-multus-conf-dir\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.033897    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wsrmr\" (UID: \"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9\") " pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.033929    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-run-netns\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.033976    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-var-lib-openvswitch\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.034001    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nz7q\" (UniqueName: \"kubernetes.io/projected/9e8945a0-1bd9-4179-8996-a165bb1db1c4-kube-api-access-7nz7q\") pod \"dns-default-gvbpd\" (UID: \"9e8945a0-1bd9-4179-8996-a165bb1db1c4\") " pod="openshift-dns/dns-default-gvbpd"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.034031    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnnxt\" (UniqueName: \"kubernetes.io/projected/d1649ad9-921c-49d9-baa5-c51f44175b8c-kube-api-access-xnnxt\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.034060    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wsrmr\" (UID: \"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9\") " pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.034092    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7e02a4de-6df2-4930-8360-4c5e09897472-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.034122    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7e02a4de-6df2-4930-8360-4c5e09897472-registration-dir\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.034148    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/801881d6-f217-4c80-a190-af953df3af04-tmp-dir\") pod \"node-resolver-hgbb5\" (UID: \"801881d6-f217-4c80-a190-af953df3af04\") " pod="openshift-dns/node-resolver-hgbb5"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.034190    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-etc-openvswitch\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.034214    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-log-socket\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.034243    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-cni-netd\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.034273    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0a5d9c55-0799-4ae1-a902-dc06f813907c-ovnkube-config\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.034301    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7e02a4de-6df2-4930-8360-4c5e09897472-socket-dir\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.034327    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b529371-cf9d-40e4-a2c5-55d69ab9a08d-host-slash\") pod \"iptables-alerter-689v8\" (UID: \"6b529371-cf9d-40e4-a2c5-55d69ab9a08d\") " pod="openshift-network-operator/iptables-alerter-689v8"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.034358    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-run-systemd\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.034387    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-cni-bin\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.034419    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4vs4\" (UniqueName: \"kubernetes.io/projected/0a5d9c55-0799-4ae1-a902-dc06f813907c-kube-api-access-s4vs4\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.034449    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e8945a0-1bd9-4179-8996-a165bb1db1c4-metrics-tls\") pod \"dns-default-gvbpd\" (UID: \"9e8945a0-1bd9-4179-8996-a165bb1db1c4\") " pod="openshift-dns/dns-default-gvbpd"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.034480    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tft2z\" (UID: \"49fbaf82-5d99-48e1-90eb-005aec9f3d1b\") " pod="openshift-insights/insights-runtime-extractor-tft2z"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.034509    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-sys\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.034542    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-cni-binary-copy\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.034573    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wsrmr\" (UID: \"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9\") " pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.034605    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-systemd-units\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.034632    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0a5d9c55-0799-4ae1-a902-dc06f813907c-ovnkube-script-lib\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.034666    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-sysconfig\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.034693    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-run\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.034724    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-tuned\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.034753    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c26dfa4-e47e-43a1-94a2-816d22fb7f3a-metrics-certs\") pod \"network-metrics-daemon-pflts\" (UID: \"7c26dfa4-e47e-43a1-94a2-816d22fb7f3a\") " pod="openshift-multus/network-metrics-daemon-pflts"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.034782    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-etc-kubernetes\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.034812    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7e02a4de-6df2-4930-8360-4c5e09897472-sys-fs\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.034842    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6b529371-cf9d-40e4-a2c5-55d69ab9a08d-iptables-alerter-script\") pod \"iptables-alerter-689v8\" (UID: \"6b529371-cf9d-40e4-a2c5-55d69ab9a08d\") " pod="openshift-network-operator/iptables-alerter-689v8"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.034872    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpfx2\" (UniqueName: \"kubernetes.io/projected/110651f3-9a70-43bf-888d-0813682bc891-kube-api-access-hpfx2\") pod \"node-ca-5drgw\" (UID: \"110651f3-9a70-43bf-888d-0813682bc891\") " pod="openshift-image-registry/node-ca-5drgw"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.034900    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-run-ovn\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.034930    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-var-lib-kubelet\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.034977    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/906c75b6-325c-4e89-8a10-e4e355884133-kubelet-config\") pod \"global-pull-secret-syncer-x9jpw\" (UID: \"906c75b6-325c-4e89-8a10-e4e355884133\") " pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.035008    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-host-run-multus-certs\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.035040    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-data-volume\") pod \"insights-runtime-extractor-tft2z\" (UID: \"49fbaf82-5d99-48e1-90eb-005aec9f3d1b\") " pod="openshift-insights/insights-runtime-extractor-tft2z"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.035066    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/906c75b6-325c-4e89-8a10-e4e355884133-dbus\") pod \"global-pull-secret-syncer-x9jpw\" (UID: \"906c75b6-325c-4e89-8a10-e4e355884133\") " pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.035096    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99jxr\" (UniqueName: \"kubernetes.io/projected/7c26dfa4-e47e-43a1-94a2-816d22fb7f3a-kube-api-access-99jxr\") pod \"network-metrics-daemon-pflts\" (UID: \"7c26dfa4-e47e-43a1-94a2-816d22fb7f3a\") " pod="openshift-multus/network-metrics-daemon-pflts"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.035132    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-cnibin\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.035165    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-host-run-netns\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.035198    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-multus-daemon-config\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.035234    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-system-cni-dir\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.035266    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-host-var-lib-cni-bin\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.035295    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-kubelet\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.035321    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-run-ovn-kubernetes\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.035354    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swvhl\" (UniqueName: \"kubernetes.io/projected/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-kube-api-access-swvhl\") pod \"insights-runtime-extractor-tft2z\" (UID: \"49fbaf82-5d99-48e1-90eb-005aec9f3d1b\") " pod="openshift-insights/insights-runtime-extractor-tft2z"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.035384    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-hostroot\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.035414    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-lib-modules\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.035443    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1649ad9-921c-49d9-baa5-c51f44175b8c-tmp\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.035470    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e1386760-3800-418f-bb1b-20e19880620b-konnectivity-ca\") pod \"konnectivity-agent-bq5p4\" (UID: \"e1386760-3800-418f-bb1b-20e19880620b\") " pod="kube-system/konnectivity-agent-bq5p4"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.035502    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/906c75b6-325c-4e89-8a10-e4e355884133-original-pull-secret\") pod \"global-pull-secret-syncer-x9jpw\" (UID: \"906c75b6-325c-4e89-8a10-e4e355884133\") " pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.035533    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-system-cni-dir\") pod \"multus-additional-cni-plugins-wsrmr\" (UID: \"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9\") " pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.035564    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvsbx\" (UniqueName: \"kubernetes.io/projected/7e02a4de-6df2-4930-8360-4c5e09897472-kube-api-access-jvsbx\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.035595    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e1386760-3800-418f-bb1b-20e19880620b-agent-certs\") pod \"konnectivity-agent-bq5p4\" (UID: \"e1386760-3800-418f-bb1b-20e19880620b\") " pod="kube-system/konnectivity-agent-bq5p4"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.035712    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-host-run-k8s-cni-cncf-io\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.035768    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-data-volume\") pod \"insights-runtime-extractor-tft2z\" (UID: \"49fbaf82-5d99-48e1-90eb-005aec9f3d1b\") " pod="openshift-insights/insights-runtime-extractor-tft2z"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.035767    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcm5z\" (UniqueName: \"kubernetes.io/projected/f6ca3dc3-231e-4a8d-a16c-7384c225110a-kube-api-access-fcm5z\") pod \"network-check-target-clx84\" (UID: \"f6ca3dc3-231e-4a8d-a16c-7384c225110a\") " pod="openshift-network-diagnostics/network-check-target-clx84"
Nov 03 08:39:37.036105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.035822    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-sysctl-conf\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:39:37.039152 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.036354    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-multus-daemon-config\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.039152 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.036444    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-cni-binary-copy\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.039152 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.036588    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bg9b\" (UniqueName: \"kubernetes.io/projected/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-kube-api-access-8bg9b\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.039152 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.036659    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-cni-binary-copy\") pod \"multus-additional-cni-plugins-wsrmr\" (UID: \"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9\") " pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:39:37.039152 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.036667    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-run-openvswitch\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.039152 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.036716    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-sysctl-d\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:39:37.039152 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.036757    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-multus-socket-dir-parent\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.039152 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.036803    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/110651f3-9a70-43bf-888d-0813682bc891-host\") pod \"node-ca-5drgw\" (UID: \"110651f3-9a70-43bf-888d-0813682bc891\") " pod="openshift-image-registry/node-ca-5drgw"
Nov 03 08:39:37.039152 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.036346    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wsrmr\" (UID: \"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9\") " pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:39:37.039152 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.036854    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0a5d9c55-0799-4ae1-a902-dc06f813907c-ovn-node-metrics-cert\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.039152 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.036890    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7329659-0fc1-4d97-b321-11e90bf7d13c-cert\") pod \"ingress-canary-74dwn\" (UID: \"d7329659-0fc1-4d97-b321-11e90bf7d13c\") " pod="openshift-ingress-canary/ingress-canary-74dwn"
Nov 03 08:39:37.039152 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.036925    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tft2z\" (UID: \"49fbaf82-5d99-48e1-90eb-005aec9f3d1b\") " pod="openshift-insights/insights-runtime-extractor-tft2z"
Nov 03 08:39:37.039152 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.036974    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/906c75b6-325c-4e89-8a10-e4e355884133-original-pull-secret\") pod \"global-pull-secret-syncer-x9jpw\" (UID: \"906c75b6-325c-4e89-8a10-e4e355884133\") " pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:39:37.039152 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.037039    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6b529371-cf9d-40e4-a2c5-55d69ab9a08d-iptables-alerter-script\") pod \"iptables-alerter-689v8\" (UID: \"6b529371-cf9d-40e4-a2c5-55d69ab9a08d\") " pod="openshift-network-operator/iptables-alerter-689v8"
Nov 03 08:39:37.039152 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.037272    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7329659-0fc1-4d97-b321-11e90bf7d13c-cert\") pod \"ingress-canary-74dwn\" (UID: \"d7329659-0fc1-4d97-b321-11e90bf7d13c\") " pod="openshift-ingress-canary/ingress-canary-74dwn"
Nov 03 08:39:37.039152 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.037283    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/801881d6-f217-4c80-a190-af953df3af04-tmp-dir\") pod \"node-resolver-hgbb5\" (UID: \"801881d6-f217-4c80-a190-af953df3af04\") " pod="openshift-dns/node-resolver-hgbb5"
Nov 03 08:39:37.039152 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.037486    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/110651f3-9a70-43bf-888d-0813682bc891-serviceca\") pod \"node-ca-5drgw\" (UID: \"110651f3-9a70-43bf-888d-0813682bc891\") " pod="openshift-image-registry/node-ca-5drgw"
Nov 03 08:39:37.039152 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.037560    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0a5d9c55-0799-4ae1-a902-dc06f813907c-ovn-node-metrics-cert\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.039152 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.037593    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0a5d9c55-0799-4ae1-a902-dc06f813907c-ovnkube-script-lib\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.039152 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.037633    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9e8945a0-1bd9-4179-8996-a165bb1db1c4-config-volume\") pod \"dns-default-gvbpd\" (UID: \"9e8945a0-1bd9-4179-8996-a165bb1db1c4\") " pod="openshift-dns/dns-default-gvbpd"
Nov 03 08:39:37.039152 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.037697    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0a5d9c55-0799-4ae1-a902-dc06f813907c-env-overrides\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.039152 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.037699    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1649ad9-921c-49d9-baa5-c51f44175b8c-tmp\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:39:37.039152 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.038134    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e8945a0-1bd9-4179-8996-a165bb1db1c4-tmp-dir\") pod \"dns-default-gvbpd\" (UID: \"9e8945a0-1bd9-4179-8996-a165bb1db1c4\") " pod="openshift-dns/dns-default-gvbpd"
Nov 03 08:39:37.039152 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.038151    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e1386760-3800-418f-bb1b-20e19880620b-konnectivity-ca\") pod \"konnectivity-agent-bq5p4\" (UID: \"e1386760-3800-418f-bb1b-20e19880620b\") " pod="kube-system/konnectivity-agent-bq5p4"
Nov 03 08:39:37.039152 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.038150    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0a5d9c55-0799-4ae1-a902-dc06f813907c-ovnkube-config\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.039152 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.038152    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-tuned\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:39:37.039152 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.038213    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e8945a0-1bd9-4179-8996-a165bb1db1c4-metrics-tls\") pod \"dns-default-gvbpd\" (UID: \"9e8945a0-1bd9-4179-8996-a165bb1db1c4\") " pod="openshift-dns/dns-default-gvbpd"
Nov 03 08:39:37.039152 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.038411    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e1386760-3800-418f-bb1b-20e19880620b-agent-certs\") pod \"konnectivity-agent-bq5p4\" (UID: \"e1386760-3800-418f-bb1b-20e19880620b\") " pod="kube-system/konnectivity-agent-bq5p4"
Nov 03 08:39:37.058028 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.058007    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 08:39:37.058028 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.058009    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 08:39:37.058184 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.058172    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 08:39:37.058234 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.058224    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 08:39:37.066266 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.066220    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tft2z\" (UID: \"49fbaf82-5d99-48e1-90eb-005aec9f3d1b\") " pod="openshift-insights/insights-runtime-extractor-tft2z"
Nov 03 08:39:37.068416 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.068399    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wsrmr\" (UID: \"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9\") " pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:39:37.086392 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.086375    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 08:39:37.086472 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.086457    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lngsh\""
Nov 03 08:39:37.086533 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.086522    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-2st7v\""
Nov 03 08:39:37.087550 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.087533    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tft2z\" (UID: \"49fbaf82-5d99-48e1-90eb-005aec9f3d1b\") " pod="openshift-insights/insights-runtime-extractor-tft2z"
Nov 03 08:39:37.093863 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.093848    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nz7q\" (UniqueName: \"kubernetes.io/projected/9e8945a0-1bd9-4179-8996-a165bb1db1c4-kube-api-access-7nz7q\") pod \"dns-default-gvbpd\" (UID: \"9e8945a0-1bd9-4179-8996-a165bb1db1c4\") " pod="openshift-dns/dns-default-gvbpd"
Nov 03 08:39:37.093973 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.093896    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnnxt\" (UniqueName: \"kubernetes.io/projected/d1649ad9-921c-49d9-baa5-c51f44175b8c-kube-api-access-xnnxt\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:39:37.094287 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.094268    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rxtf\" (UniqueName: \"kubernetes.io/projected/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-kube-api-access-7rxtf\") pod \"multus-additional-cni-plugins-wsrmr\" (UID: \"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9\") " pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:39:37.094696 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.094680    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcm5z\" (UniqueName: \"kubernetes.io/projected/f6ca3dc3-231e-4a8d-a16c-7384c225110a-kube-api-access-fcm5z\") pod \"network-check-target-clx84\" (UID: \"f6ca3dc3-231e-4a8d-a16c-7384c225110a\") " pod="openshift-network-diagnostics/network-check-target-clx84"
Nov 03 08:39:37.094741 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.094723    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99jxr\" (UniqueName: \"kubernetes.io/projected/7c26dfa4-e47e-43a1-94a2-816d22fb7f3a-kube-api-access-99jxr\") pod \"network-metrics-daemon-pflts\" (UID: \"7c26dfa4-e47e-43a1-94a2-816d22fb7f3a\") " pod="openshift-multus/network-metrics-daemon-pflts"
Nov 03 08:39:37.094773 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.094727    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njvj8\" (UniqueName: \"kubernetes.io/projected/801881d6-f217-4c80-a190-af953df3af04-kube-api-access-njvj8\") pod \"node-resolver-hgbb5\" (UID: \"801881d6-f217-4c80-a190-af953df3af04\") " pod="openshift-dns/node-resolver-hgbb5"
Nov 03 08:39:37.094809 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.094799    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4vs4\" (UniqueName: \"kubernetes.io/projected/0a5d9c55-0799-4ae1-a902-dc06f813907c-kube-api-access-s4vs4\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.095728 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.095712    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv94t\" (UniqueName: \"kubernetes.io/projected/d7329659-0fc1-4d97-b321-11e90bf7d13c-kube-api-access-kv94t\") pod \"ingress-canary-74dwn\" (UID: \"d7329659-0fc1-4d97-b321-11e90bf7d13c\") " pod="openshift-ingress-canary/ingress-canary-74dwn"
Nov 03 08:39:37.096241 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.096225    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvsbx\" (UniqueName: \"kubernetes.io/projected/7e02a4de-6df2-4930-8360-4c5e09897472-kube-api-access-jvsbx\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:39:37.096427 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.096409    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgfkn\" (UniqueName: \"kubernetes.io/projected/6b529371-cf9d-40e4-a2c5-55d69ab9a08d-kube-api-access-sgfkn\") pod \"iptables-alerter-689v8\" (UID: \"6b529371-cf9d-40e4-a2c5-55d69ab9a08d\") " pod="openshift-network-operator/iptables-alerter-689v8"
Nov 03 08:39:37.096473 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.096424    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bg9b\" (UniqueName: \"kubernetes.io/projected/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-kube-api-access-8bg9b\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.096473 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.096414    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swvhl\" (UniqueName: \"kubernetes.io/projected/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-kube-api-access-swvhl\") pod \"insights-runtime-extractor-tft2z\" (UID: \"49fbaf82-5d99-48e1-90eb-005aec9f3d1b\") " pod="openshift-insights/insights-runtime-extractor-tft2z"
Nov 03 08:39:37.096473 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.096459    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpfx2\" (UniqueName: \"kubernetes.io/projected/110651f3-9a70-43bf-888d-0813682bc891-kube-api-access-hpfx2\") pod \"node-ca-5drgw\" (UID: \"110651f3-9a70-43bf-888d-0813682bc891\") " pod="openshift-image-registry/node-ca-5drgw"
Nov 03 08:39:37.137674 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.137650    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-slash\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.137753 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.137676    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/801881d6-f217-4c80-a190-af953df3af04-hosts-file\") pod \"node-resolver-hgbb5\" (UID: \"801881d6-f217-4c80-a190-af953df3af04\") " pod="openshift-dns/node-resolver-hgbb5"
Nov 03 08:39:37.137753 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.137689    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-node-log\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.137753 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.137704    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.137753 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.137721    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-crio-socket\") pod \"insights-runtime-extractor-tft2z\" (UID: \"49fbaf82-5d99-48e1-90eb-005aec9f3d1b\") " pod="openshift-insights/insights-runtime-extractor-tft2z"
Nov 03 08:39:37.137753 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.137734    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-host-var-lib-kubelet\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.137753 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.137748    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-multus-cni-dir\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.138054 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.137767    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-multus-conf-dir\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.138054 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.137773    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/801881d6-f217-4c80-a190-af953df3af04-hosts-file\") pod \"node-resolver-hgbb5\" (UID: \"801881d6-f217-4c80-a190-af953df3af04\") " pod="openshift-dns/node-resolver-hgbb5"
Nov 03 08:39:37.138054 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.137782    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.138054 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.137788    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-run-netns\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.138054 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.137805    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/49fbaf82-5d99-48e1-90eb-005aec9f3d1b-crio-socket\") pod \"insights-runtime-extractor-tft2z\" (UID: \"49fbaf82-5d99-48e1-90eb-005aec9f3d1b\") " pod="openshift-insights/insights-runtime-extractor-tft2z"
Nov 03 08:39:37.138054 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.137811    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-node-log\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.138054 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.137823    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-run-netns\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.138054 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.137829    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-var-lib-openvswitch\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.138054 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.137825    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-multus-cni-dir\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.138054 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.137836    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-multus-conf-dir\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.138054 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.137773    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-slash\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.138054 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.137861    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wsrmr\" (UID: \"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9\") " pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:39:37.138054 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.137885    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7e02a4de-6df2-4930-8360-4c5e09897472-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:39:37.138054 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.137882    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-var-lib-openvswitch\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.138054 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.137908    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7e02a4de-6df2-4930-8360-4c5e09897472-registration-dir\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:39:37.138054 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.137930    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-host-var-lib-kubelet\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.138054 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.137934    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-etc-openvswitch\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.138054 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.137979    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-etc-openvswitch\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.138054 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.137981    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7e02a4de-6df2-4930-8360-4c5e09897472-registration-dir\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:39:37.138054 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.137980    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7e02a4de-6df2-4930-8360-4c5e09897472-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:39:37.138054 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.137985    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-log-socket\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.138054 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.137899    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wsrmr\" (UID: \"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9\") " pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:39:37.138054 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138012    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-log-socket\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.138054 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138037    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-cni-netd\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138069    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7e02a4de-6df2-4930-8360-4c5e09897472-socket-dir\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138074    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-cni-netd\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138098    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b529371-cf9d-40e4-a2c5-55d69ab9a08d-host-slash\") pod \"iptables-alerter-689v8\" (UID: \"6b529371-cf9d-40e4-a2c5-55d69ab9a08d\") " pod="openshift-network-operator/iptables-alerter-689v8"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138121    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7e02a4de-6df2-4930-8360-4c5e09897472-socket-dir\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138146    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-run-systemd\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138160    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-cni-bin\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138200    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-cni-bin\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138212    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-sys\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138219    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-run-systemd\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138228    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-systemd-units\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138200    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b529371-cf9d-40e4-a2c5-55d69ab9a08d-host-slash\") pod \"iptables-alerter-689v8\" (UID: \"6b529371-cf9d-40e4-a2c5-55d69ab9a08d\") " pod="openshift-network-operator/iptables-alerter-689v8"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138239    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-sys\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138249    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-sysconfig\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138262    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-run\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138271    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-systemd-units\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138300    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-sysconfig\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138303    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-run\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138275    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c26dfa4-e47e-43a1-94a2-816d22fb7f3a-metrics-certs\") pod \"network-metrics-daemon-pflts\" (UID: \"7c26dfa4-e47e-43a1-94a2-816d22fb7f3a\") " pod="openshift-multus/network-metrics-daemon-pflts"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138332    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-etc-kubernetes\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138347    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7e02a4de-6df2-4930-8360-4c5e09897472-sys-fs\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138368    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-run-ovn\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138383    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-var-lib-kubelet\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138399    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/906c75b6-325c-4e89-8a10-e4e355884133-kubelet-config\") pod \"global-pull-secret-syncer-x9jpw\" (UID: \"906c75b6-325c-4e89-8a10-e4e355884133\") " pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138414    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-host-run-multus-certs\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138420    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7e02a4de-6df2-4930-8360-4c5e09897472-sys-fs\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138429    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/906c75b6-325c-4e89-8a10-e4e355884133-dbus\") pod \"global-pull-secret-syncer-x9jpw\" (UID: \"906c75b6-325c-4e89-8a10-e4e355884133\") " pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138434    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-etc-kubernetes\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138447    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-run-ovn\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138472    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-host-run-multus-certs\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138480    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/906c75b6-325c-4e89-8a10-e4e355884133-dbus\") pod \"global-pull-secret-syncer-x9jpw\" (UID: \"906c75b6-325c-4e89-8a10-e4e355884133\") " pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138477    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/906c75b6-325c-4e89-8a10-e4e355884133-kubelet-config\") pod \"global-pull-secret-syncer-x9jpw\" (UID: \"906c75b6-325c-4e89-8a10-e4e355884133\") " pod="kube-system/global-pull-secret-syncer-x9jpw"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138494    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-cnibin\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138510    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-host-run-netns\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138526    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-system-cni-dir\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138481    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-var-lib-kubelet\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138543    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-host-var-lib-cni-bin\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138567    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-kubelet\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138571    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-host-run-netns\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138583    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-system-cni-dir\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138583    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-host-var-lib-cni-bin\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138593    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-run-ovn-kubernetes\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138561    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-cnibin\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138614    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-kubelet\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138613    4473 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138617    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-hostroot\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138636    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-host-run-ovn-kubernetes\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138649    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-lib-modules\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138638    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-hostroot\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138683    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-system-cni-dir\") pod \"multus-additional-cni-plugins-wsrmr\" (UID: \"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9\") " pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138706    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-host-run-k8s-cni-cncf-io\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138726    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-sysctl-conf\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138750    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-host-run-k8s-cni-cncf-io\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138752    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-lib-modules\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138751    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-run-openvswitch\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138729    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-system-cni-dir\") pod \"multus-additional-cni-plugins-wsrmr\" (UID: \"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9\") " pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138783    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-sysctl-d\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138788    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0a5d9c55-0799-4ae1-a902-dc06f813907c-run-openvswitch\") pod \"ovnkube-node-bt4zt\" (UID: \"0a5d9c55-0799-4ae1-a902-dc06f813907c\") " pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138806    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-multus-socket-dir-parent\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138825    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/110651f3-9a70-43bf-888d-0813682bc891-host\") pod \"node-ca-5drgw\" (UID: \"110651f3-9a70-43bf-888d-0813682bc891\") " pod="openshift-image-registry/node-ca-5drgw"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138847    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-modprobe-d\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138869    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-kubernetes\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138880    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/110651f3-9a70-43bf-888d-0813682bc891-host\") pod \"node-ca-5drgw\" (UID: \"110651f3-9a70-43bf-888d-0813682bc891\") " pod="openshift-image-registry/node-ca-5drgw"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138889    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-host-var-lib-cni-multus\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138901    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-sysctl-conf\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138880    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-multus-socket-dir-parent\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138913    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-cnibin\") pod \"multus-additional-cni-plugins-wsrmr\" (UID: \"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9\") " pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138919    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-kubernetes\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138922    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/105e26de-f1f1-4aae-99ef-e6a9491ccbe8-host-var-lib-cni-multus\") pod \"multus-kzcqg\" (UID: \"105e26de-f1f1-4aae-99ef-e6a9491ccbe8\") " pod="openshift-multus/multus-kzcqg"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138961    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7e02a4de-6df2-4930-8360-4c5e09897472-etc-selinux\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138982    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9-cnibin\") pod \"multus-additional-cni-plugins-wsrmr\" (UID: \"968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9\") " pod="openshift-multus/multus-additional-cni-plugins-wsrmr"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138991    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-systemd\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.138964    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-sysctl-d\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.139011    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7e02a4de-6df2-4930-8360-4c5e09897472-etc-selinux\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.139046    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-systemd\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.139051    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-host\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:39:37.139017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.139012    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-host\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:39:37.141549 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.139069    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d1649ad9-921c-49d9-baa5-c51f44175b8c-etc-modprobe-d\") pod \"tuned-jpll8\" (UID: \"d1649ad9-921c-49d9-baa5-c51f44175b8c\") " pod="openshift-cluster-node-tuning-operator/tuned-jpll8"
Nov 03 08:39:37.141549 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.139212    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7e02a4de-6df2-4930-8360-4c5e09897472-device-dir\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:39:37.141549 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.139272    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7e02a4de-6df2-4930-8360-4c5e09897472-device-dir\") pod \"aws-ebs-csi-driver-node-l8cfs\" (UID: \"7e02a4de-6df2-4930-8360-4c5e09897472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8cfs"
Nov 03 08:39:37.141549 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.140739    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c26dfa4-e47e-43a1-94a2-816d22fb7f3a-metrics-certs\") pod \"network-metrics-daemon-pflts\" (UID: \"7c26dfa4-e47e-43a1-94a2-816d22fb7f3a\") " pod="openshift-multus/network-metrics-daemon-pflts"
Nov 03 08:39:37.181874 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.181850    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bq5p4_e1386760-3800-418f-bb1b-20e19880620b/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:37.227410 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.227384    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pflts"
Nov 03 08:39:37.229037 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.229022    4473 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 08:39:37.371549 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.371522    4473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bq5p4_e1386760-3800-418f-bb1b-20e19880620b/konnectivity-agent/0.log" limit=1024
Nov 03 08:39:37.620356 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.620331    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pflts"]
Nov 03 08:39:37.623867 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:37.623841    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c26dfa4_e47e_43a1_94a2_816d22fb7f3a.slice/crio-dde9d826b78152040e5468805e59dfbed7b08584ff5fe0e1926c875c199c1795 WatchSource:0}: Error finding container dde9d826b78152040e5468805e59dfbed7b08584ff5fe0e1926c875c199c1795: Status 404 returned error can't find the container with id dde9d826b78152040e5468805e59dfbed7b08584ff5fe0e1926c875c199c1795
Nov 03 08:39:37.900716 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:37.900623    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pflts" event={"ID":"7c26dfa4-e47e-43a1-94a2-816d22fb7f3a","Type":"ContainerStarted","Data":"dde9d826b78152040e5468805e59dfbed7b08584ff5fe0e1926c875c199c1795"}
Nov 03 08:39:38.905637 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:38.905599    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tft2z" event={"ID":"49fbaf82-5d99-48e1-90eb-005aec9f3d1b","Type":"ContainerStarted","Data":"8cf510f9adbc530bc4d45ebbf5a9f0ff1feb23c05e1d93eaaace80c8912f132b"}
Nov 03 08:39:39.911449 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:39.911360    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pflts" event={"ID":"7c26dfa4-e47e-43a1-94a2-816d22fb7f3a","Type":"ContainerStarted","Data":"8742c7c7fdcd8ad3949955e593602c3e860820e5078714e320af07be0799f435"}
Nov 03 08:39:39.911449 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:39.911426    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pflts" event={"ID":"7c26dfa4-e47e-43a1-94a2-816d22fb7f3a","Type":"ContainerStarted","Data":"d15046c6b9373b890e570095d80dc3e88e5b8a7b537af568da25735a61b69179"}
Nov 03 08:39:39.913678 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:39.913648    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gvbpd" event={"ID":"9e8945a0-1bd9-4179-8996-a165bb1db1c4","Type":"ContainerStarted","Data":"3daa48d1820dbb523a41bebb03149cfc66b4aa271b74b6a50fecf457b6ff3be0"}
Nov 03 08:39:39.913813 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:39.913706    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gvbpd" event={"ID":"9e8945a0-1bd9-4179-8996-a165bb1db1c4","Type":"ContainerStarted","Data":"20b128343006016bd5f693c11ad2b04e7ea1ba5364b786a8ca0f857f57edbd2f"}
Nov 03 08:39:40.116091 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:40.116056    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gvbpd"
Nov 03 08:39:40.606680 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:40.606607    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:40.919519 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:40.919479    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tft2z" event={"ID":"49fbaf82-5d99-48e1-90eb-005aec9f3d1b","Type":"ContainerStarted","Data":"8aa9fdd7e6c1ac4c231244c0725dfddfc110deb39eb37abf5a7c593b0521f9b7"}
Nov 03 08:39:43.106044 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:43.106012    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:43.106395 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:43.106267    4473 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 08:39:46.485963 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:46.485916    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-clx84"
Nov 03 08:39:46.935621 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:39:46.935591    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:39:47.608254 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:47.608229    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-pbpxk"]
Nov 03 08:39:47.609749 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:47.609733    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-pbpxk"
Nov 03 08:39:47.683298 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:47.683274    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-l6j2j\""
Nov 03 08:39:47.700776 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:47.700755    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 03 08:39:47.700881 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:47.700777    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ae525063-a06b-4860-af1a-3bcd116858c6-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-pbpxk\" (UID: \"ae525063-a06b-4860-af1a-3bcd116858c6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-pbpxk"
Nov 03 08:39:47.712199 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:47.712176    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-pbpxk"]
Nov 03 08:39:47.802071 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:47.802039    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ae525063-a06b-4860-af1a-3bcd116858c6-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-pbpxk\" (UID: \"ae525063-a06b-4860-af1a-3bcd116858c6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-pbpxk"
Nov 03 08:39:47.804714 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:47.804694    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ae525063-a06b-4860-af1a-3bcd116858c6-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-pbpxk\" (UID: \"ae525063-a06b-4860-af1a-3bcd116858c6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-pbpxk"
Nov 03 08:39:47.842892 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:47.842873    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bt4zt"
Nov 03 08:39:47.919052 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:47.919028    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-pbpxk"
Nov 03 08:39:48.284606 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:48.284539    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-pbpxk"]
Nov 03 08:39:48.287398 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:48.287370    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae525063_a06b_4860_af1a_3bcd116858c6.slice/crio-cf556a80a5568cc6b266f1a4fd828e3c5f5dad6d10b31a39912980bf9a1d6a85 WatchSource:0}: Error finding container cf556a80a5568cc6b266f1a4fd828e3c5f5dad6d10b31a39912980bf9a1d6a85: Status 404 returned error can't find the container with id cf556a80a5568cc6b266f1a4fd828e3c5f5dad6d10b31a39912980bf9a1d6a85
Nov 03 08:39:48.942771 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:48.942732    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-pbpxk" event={"ID":"ae525063-a06b-4860-af1a-3bcd116858c6","Type":"ContainerStarted","Data":"cf556a80a5568cc6b266f1a4fd828e3c5f5dad6d10b31a39912980bf9a1d6a85"}
Nov 03 08:39:49.922362 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:49.922281    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gvbpd"
Nov 03 08:39:49.947135 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:49.947104    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-pbpxk" event={"ID":"ae525063-a06b-4860-af1a-3bcd116858c6","Type":"ContainerStarted","Data":"3f08b6f3d5617499b480c6eb528648d0c4dc9c03495dff05369224d6c87d926a"}
Nov 03 08:39:49.947477 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:49.947156    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-pbpxk"
Nov 03 08:39:49.951517 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:49.951496    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-pbpxk"
Nov 03 08:39:50.155323 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:50.155268    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-pbpxk" podStartSLOduration=1.7776595039999998 podStartE2EDuration="3.155252992s" totalImagesPullingTime="1.377593488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:50.154786425 +0000 UTC m=+14.423849879" watchObservedRunningTime="2026-11-03 08:39:50.155252992 +0000 UTC m=+14.424316425"
Nov 03 08:39:50.831145 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:50.831109    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-qrl96"]
Nov 03 08:39:50.832831 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:50.832815    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qrl96"
Nov 03 08:39:50.920216 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:50.920188    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b6b8ee0a-1be5-40c0-8878-4eba660f6309-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-qrl96\" (UID: \"b6b8ee0a-1be5-40c0-8878-4eba660f6309\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qrl96"
Nov 03 08:39:50.920216 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:50.920220    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6b8ee0a-1be5-40c0-8878-4eba660f6309-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-qrl96\" (UID: \"b6b8ee0a-1be5-40c0-8878-4eba660f6309\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qrl96"
Nov 03 08:39:50.920445 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:50.920243    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsmjt\" (UniqueName: \"kubernetes.io/projected/b6b8ee0a-1be5-40c0-8878-4eba660f6309-kube-api-access-nsmjt\") pod \"prometheus-operator-9cd89b74c-qrl96\" (UID: \"b6b8ee0a-1be5-40c0-8878-4eba660f6309\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qrl96"
Nov 03 08:39:50.920445 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:50.920275    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6b8ee0a-1be5-40c0-8878-4eba660f6309-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-qrl96\" (UID: \"b6b8ee0a-1be5-40c0-8878-4eba660f6309\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qrl96"
Nov 03 08:39:50.936553 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:50.936524    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 03 08:39:50.936553 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:50.936536    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-hzksr\""
Nov 03 08:39:50.936745 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:50.936562    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 08:39:50.936745 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:50.936563    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 08:39:50.936745 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:50.936602    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 03 08:39:50.937529 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:50.937514    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 08:39:50.987697 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:50.987668    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-qrl96"]
Nov 03 08:39:51.021639 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:51.021604    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6b8ee0a-1be5-40c0-8878-4eba660f6309-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-qrl96\" (UID: \"b6b8ee0a-1be5-40c0-8878-4eba660f6309\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qrl96"
Nov 03 08:39:51.021760 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:51.021654    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b6b8ee0a-1be5-40c0-8878-4eba660f6309-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-qrl96\" (UID: \"b6b8ee0a-1be5-40c0-8878-4eba660f6309\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qrl96"
Nov 03 08:39:51.021760 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:51.021683    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6b8ee0a-1be5-40c0-8878-4eba660f6309-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-qrl96\" (UID: \"b6b8ee0a-1be5-40c0-8878-4eba660f6309\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qrl96"
Nov 03 08:39:51.021760 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:51.021721    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsmjt\" (UniqueName: \"kubernetes.io/projected/b6b8ee0a-1be5-40c0-8878-4eba660f6309-kube-api-access-nsmjt\") pod \"prometheus-operator-9cd89b74c-qrl96\" (UID: \"b6b8ee0a-1be5-40c0-8878-4eba660f6309\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qrl96"
Nov 03 08:39:51.022349 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:51.022326    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6b8ee0a-1be5-40c0-8878-4eba660f6309-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-qrl96\" (UID: \"b6b8ee0a-1be5-40c0-8878-4eba660f6309\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qrl96"
Nov 03 08:39:51.024327 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:51.024304    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6b8ee0a-1be5-40c0-8878-4eba660f6309-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-qrl96\" (UID: \"b6b8ee0a-1be5-40c0-8878-4eba660f6309\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qrl96"
Nov 03 08:39:51.024417 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:51.024363    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b6b8ee0a-1be5-40c0-8878-4eba660f6309-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-qrl96\" (UID: \"b6b8ee0a-1be5-40c0-8878-4eba660f6309\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qrl96"
Nov 03 08:39:51.082989 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:51.082913    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsmjt\" (UniqueName: \"kubernetes.io/projected/b6b8ee0a-1be5-40c0-8878-4eba660f6309-kube-api-access-nsmjt\") pod \"prometheus-operator-9cd89b74c-qrl96\" (UID: \"b6b8ee0a-1be5-40c0-8878-4eba660f6309\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qrl96"
Nov 03 08:39:51.143953 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:51.143929    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qrl96"
Nov 03 08:39:51.512008 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:51.511885    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-qrl96"]
Nov 03 08:39:51.514353 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:51.514328    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b8ee0a_1be5_40c0_8878_4eba660f6309.slice/crio-5ec39d13c47e12f1e2414b21ca8f20ba2e63edef27a78ff1cca38743415fe3b1 WatchSource:0}: Error finding container 5ec39d13c47e12f1e2414b21ca8f20ba2e63edef27a78ff1cca38743415fe3b1: Status 404 returned error can't find the container with id 5ec39d13c47e12f1e2414b21ca8f20ba2e63edef27a78ff1cca38743415fe3b1
Nov 03 08:39:51.953312 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:51.953275    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qrl96" event={"ID":"b6b8ee0a-1be5-40c0-8878-4eba660f6309","Type":"ContainerStarted","Data":"5ec39d13c47e12f1e2414b21ca8f20ba2e63edef27a78ff1cca38743415fe3b1"}
Nov 03 08:39:53.965423 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:53.965380    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qrl96" event={"ID":"b6b8ee0a-1be5-40c0-8878-4eba660f6309","Type":"ContainerStarted","Data":"9b63cded548b141fb7151ef10088ca4753044ce37aaec00a32757de69fa2e1fb"}
Nov 03 08:39:53.965898 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:53.965473    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qrl96" event={"ID":"b6b8ee0a-1be5-40c0-8878-4eba660f6309","Type":"ContainerStarted","Data":"bd3eb06b6c8940a21bcf5f6934ec623885341e30e4b0ca975d45f47054bedad4"}
Nov 03 08:39:54.082713 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:54.082662    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qrl96" podStartSLOduration=2.554474663 podStartE2EDuration="4.082648697s" totalImagesPullingTime="1.528174034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:39:54.08185986 +0000 UTC m=+18.350923310" watchObservedRunningTime="2026-11-03 08:39:54.082648697 +0000 UTC m=+18.351712130"
Nov 03 08:39:57.532069 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:57.532040    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hjsfs"]
Nov 03 08:39:57.534162 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:57.534147    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hjsfs"
Nov 03 08:39:57.562086 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:57.562058    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9-root\") pod \"node-exporter-hjsfs\" (UID: \"ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9\") " pod="openshift-monitoring/node-exporter-hjsfs"
Nov 03 08:39:57.562086 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:57.562088    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9-node-exporter-textfile\") pod \"node-exporter-hjsfs\" (UID: \"ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9\") " pod="openshift-monitoring/node-exporter-hjsfs"
Nov 03 08:39:57.562236 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:57.562103    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9-node-exporter-tls\") pod \"node-exporter-hjsfs\" (UID: \"ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9\") " pod="openshift-monitoring/node-exporter-hjsfs"
Nov 03 08:39:57.562236 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:57.562117    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9-node-exporter-wtmp\") pod \"node-exporter-hjsfs\" (UID: \"ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9\") " pod="openshift-monitoring/node-exporter-hjsfs"
Nov 03 08:39:57.562236 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:57.562147    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-894nn\" (UniqueName: \"kubernetes.io/projected/ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9-kube-api-access-894nn\") pod \"node-exporter-hjsfs\" (UID: \"ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9\") " pod="openshift-monitoring/node-exporter-hjsfs"
Nov 03 08:39:57.562236 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:57.562201    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9-sys\") pod \"node-exporter-hjsfs\" (UID: \"ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9\") " pod="openshift-monitoring/node-exporter-hjsfs"
Nov 03 08:39:57.562375 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:57.562242    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hjsfs\" (UID: \"ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9\") " pod="openshift-monitoring/node-exporter-hjsfs"
Nov 03 08:39:57.562375 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:57.562308    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9-metrics-client-ca\") pod \"node-exporter-hjsfs\" (UID: \"ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9\") " pod="openshift-monitoring/node-exporter-hjsfs"
Nov 03 08:39:57.562375 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:57.562344    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9-node-exporter-accelerators-collector-config\") pod \"node-exporter-hjsfs\" (UID: \"ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9\") " pod="openshift-monitoring/node-exporter-hjsfs"
Nov 03 08:39:57.662609 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:57.662572    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9-metrics-client-ca\") pod \"node-exporter-hjsfs\" (UID: \"ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9\") " pod="openshift-monitoring/node-exporter-hjsfs"
Nov 03 08:39:57.662609 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:57.662606    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9-node-exporter-accelerators-collector-config\") pod \"node-exporter-hjsfs\" (UID: \"ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9\") " pod="openshift-monitoring/node-exporter-hjsfs"
Nov 03 08:39:57.662787 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:57.662645    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9-root\") pod \"node-exporter-hjsfs\" (UID: \"ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9\") " pod="openshift-monitoring/node-exporter-hjsfs"
Nov 03 08:39:57.662787 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:57.662692    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9-root\") pod \"node-exporter-hjsfs\" (UID: \"ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9\") " pod="openshift-monitoring/node-exporter-hjsfs"
Nov 03 08:39:57.662864 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:57.662794    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9-node-exporter-textfile\") pod \"node-exporter-hjsfs\" (UID: \"ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9\") " pod="openshift-monitoring/node-exporter-hjsfs"
Nov 03 08:39:57.662864 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:57.662823    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9-node-exporter-tls\") pod \"node-exporter-hjsfs\" (UID: \"ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9\") " pod="openshift-monitoring/node-exporter-hjsfs"
Nov 03 08:39:57.662864 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:57.662839    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9-node-exporter-wtmp\") pod \"node-exporter-hjsfs\" (UID: \"ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9\") " pod="openshift-monitoring/node-exporter-hjsfs"
Nov 03 08:39:57.662864 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:57.662862    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-894nn\" (UniqueName: \"kubernetes.io/projected/ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9-kube-api-access-894nn\") pod \"node-exporter-hjsfs\" (UID: \"ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9\") " pod="openshift-monitoring/node-exporter-hjsfs"
Nov 03 08:39:57.663052 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:57.662996    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9-sys\") pod \"node-exporter-hjsfs\" (UID: \"ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9\") " pod="openshift-monitoring/node-exporter-hjsfs"
Nov 03 08:39:57.663052 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:57.663033    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9-sys\") pod \"node-exporter-hjsfs\" (UID: \"ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9\") " pod="openshift-monitoring/node-exporter-hjsfs"
Nov 03 08:39:57.663052 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:57.663038    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hjsfs\" (UID: \"ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9\") " pod="openshift-monitoring/node-exporter-hjsfs"
Nov 03 08:39:57.663052 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:57.662999    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9-node-exporter-wtmp\") pod \"node-exporter-hjsfs\" (UID: \"ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9\") " pod="openshift-monitoring/node-exporter-hjsfs"
Nov 03 08:39:57.663196 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:57.663082    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9-node-exporter-textfile\") pod \"node-exporter-hjsfs\" (UID: \"ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9\") " pod="openshift-monitoring/node-exporter-hjsfs"
Nov 03 08:39:57.663260 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:57.663244    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9-metrics-client-ca\") pod \"node-exporter-hjsfs\" (UID: \"ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9\") " pod="openshift-monitoring/node-exporter-hjsfs"
Nov 03 08:39:57.703037 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:57.703013    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 08:39:57.703160 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:57.703065    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 08:39:57.703160 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:57.703154    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-h97f4\""
Nov 03 08:39:57.703303 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:57.703209    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 08:39:57.705513 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:57.705494    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hjsfs\" (UID: \"ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9\") " pod="openshift-monitoring/node-exporter-hjsfs"
Nov 03 08:39:57.713781 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:57.713761    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9-node-exporter-accelerators-collector-config\") pod \"node-exporter-hjsfs\" (UID: \"ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9\") " pod="openshift-monitoring/node-exporter-hjsfs"
Nov 03 08:39:57.715405 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:57.715389    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9-node-exporter-tls\") pod \"node-exporter-hjsfs\" (UID: \"ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9\") " pod="openshift-monitoring/node-exporter-hjsfs"
Nov 03 08:39:57.721788 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:57.721762    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-894nn\" (UniqueName: \"kubernetes.io/projected/ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9-kube-api-access-894nn\") pod \"node-exporter-hjsfs\" (UID: \"ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9\") " pod="openshift-monitoring/node-exporter-hjsfs"
Nov 03 08:39:57.845341 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:57.845275    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hjsfs"
Nov 03 08:39:57.855807 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:39:57.855765    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba36bbf3_f9d7_4f0b_98af_ee5dc6fe87e9.slice/crio-98fd59b1c50c24887100340436dd019ca047a77d8452672cbeb012595f95f129 WatchSource:0}: Error finding container 98fd59b1c50c24887100340436dd019ca047a77d8452672cbeb012595f95f129: Status 404 returned error can't find the container with id 98fd59b1c50c24887100340436dd019ca047a77d8452672cbeb012595f95f129
Nov 03 08:39:57.980745 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:57.980711    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hjsfs" event={"ID":"ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9","Type":"ContainerStarted","Data":"98fd59b1c50c24887100340436dd019ca047a77d8452672cbeb012595f95f129"}
Nov 03 08:39:58.340506 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:58.340477    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bq5p4"
Nov 03 08:39:59.830255 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.830222    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 08:39:59.832384 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.832368    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:39:59.877593 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.877560    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84d30531-a4d7-43e8-b283-4ce98cf06b9d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:39:59.877593 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.877597    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:39:59.877774 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.877618    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/84d30531-a4d7-43e8-b283-4ce98cf06b9d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:39:59.877774 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.877714    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bl8g\" (UniqueName: \"kubernetes.io/projected/84d30531-a4d7-43e8-b283-4ce98cf06b9d-kube-api-access-2bl8g\") pod \"alertmanager-main-0\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:39:59.877774 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.877741    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84d30531-a4d7-43e8-b283-4ce98cf06b9d-config-out\") pod \"alertmanager-main-0\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:39:59.877774 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.877770    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:39:59.877939 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.877792    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:39:59.877939 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.877830    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84d30531-a4d7-43e8-b283-4ce98cf06b9d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:39:59.878023 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.877959    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-web-config\") pod \"alertmanager-main-0\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:39:59.878023 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.877976    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:39:59.878023 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.877995    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84d30531-a4d7-43e8-b283-4ce98cf06b9d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:39:59.878023 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.878010    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:39:59.878165 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.878055    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-config-volume\") pod \"alertmanager-main-0\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:39:59.937413 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.937337    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 08:39:59.937413 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.937341    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 08:39:59.937413 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.937383    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 08:39:59.937413 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.937390    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 08:39:59.937413 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.937418    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 08:39:59.937764 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.937428    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 08:39:59.937764 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.937436    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 08:39:59.937764 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.937569    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-g2c94\""
Nov 03 08:39:59.938531 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.938513    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 08:39:59.978512 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.978482    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:39:59.978675 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.978521    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/84d30531-a4d7-43e8-b283-4ce98cf06b9d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:39:59.978675 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.978552    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bl8g\" (UniqueName: \"kubernetes.io/projected/84d30531-a4d7-43e8-b283-4ce98cf06b9d-kube-api-access-2bl8g\") pod \"alertmanager-main-0\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:39:59.978675 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.978582    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84d30531-a4d7-43e8-b283-4ce98cf06b9d-config-out\") pod \"alertmanager-main-0\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:39:59.978675 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.978619    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:39:59.978675 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.978651    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:39:59.979036 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.978684    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84d30531-a4d7-43e8-b283-4ce98cf06b9d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:39:59.979036 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.978711    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-web-config\") pod \"alertmanager-main-0\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:39:59.979036 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.978737    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:39:59.979036 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.978863    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84d30531-a4d7-43e8-b283-4ce98cf06b9d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:39:59.979036 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.978890    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:39:59.979036 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.978927    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-config-volume\") pod \"alertmanager-main-0\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:39:59.979036 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.978975    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84d30531-a4d7-43e8-b283-4ce98cf06b9d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:39:59.980518 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.980112    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84d30531-a4d7-43e8-b283-4ce98cf06b9d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:39:59.981244 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.981033    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/84d30531-a4d7-43e8-b283-4ce98cf06b9d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:39:59.983565 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.983522    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84d30531-a4d7-43e8-b283-4ce98cf06b9d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:39:59.983707 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.983682    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:39:59.984025 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.983939    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84d30531-a4d7-43e8-b283-4ce98cf06b9d-config-out\") pod \"alertmanager-main-0\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:39:59.984135 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.984116    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:39:59.984191 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.984113    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:39:59.984248 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.984224    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:39:59.984713 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.984668    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:39:59.985221 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.985201    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-web-config\") pod \"alertmanager-main-0\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:39:59.988560 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.988544    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 08:39:59.989958 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.989916    4473 generic.go:356] "Generic (PLEG): container finished" podID="ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9" containerID="2c0f9dc8e3abeb9ac33d7b8d22a40db686bc6c3625a41ac7420add410be4142b" exitCode=0
Nov 03 08:39:59.990053 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.989975    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hjsfs" event={"ID":"ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9","Type":"ContainerDied","Data":"2c0f9dc8e3abeb9ac33d7b8d22a40db686bc6c3625a41ac7420add410be4142b"}
Nov 03 08:39:59.992357 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:39:59.992331    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84d30531-a4d7-43e8-b283-4ce98cf06b9d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:00.038428 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.038404    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 08:40:00.042010 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.041986    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-config-volume\") pod \"alertmanager-main-0\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:00.042770 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.042746    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bl8g\" (UniqueName: \"kubernetes.io/projected/84d30531-a4d7-43e8-b283-4ce98cf06b9d-kube-api-access-2bl8g\") pod \"alertmanager-main-0\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:00.147279 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.147244    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:00.189292 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.189254    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-c5548d8cc-xj2td"]
Nov 03 08:40:00.196300 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.196275    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-c5548d8cc-xj2td"
Nov 03 08:40:00.252818 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.252666    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 03 08:40:00.252818 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.252734    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 03 08:40:00.252818 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.252783    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 03 08:40:00.252818 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.252806    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 03 08:40:00.253182 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.252912    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-decjprmlrlhe0\""
Nov 03 08:40:00.253182 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.252989    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-vz6zg\""
Nov 03 08:40:00.253182 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.253143    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-c5548d8cc-xj2td"]
Nov 03 08:40:00.253453 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.253436    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 03 08:40:00.281143 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.281114    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b030cb9d-3c84-4482-9761-a507e84c9204-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c5548d8cc-xj2td\" (UID: \"b030cb9d-3c84-4482-9761-a507e84c9204\") " pod="openshift-monitoring/thanos-querier-c5548d8cc-xj2td"
Nov 03 08:40:00.281331 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.281150    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b030cb9d-3c84-4482-9761-a507e84c9204-secret-thanos-querier-tls\") pod \"thanos-querier-c5548d8cc-xj2td\" (UID: \"b030cb9d-3c84-4482-9761-a507e84c9204\") " pod="openshift-monitoring/thanos-querier-c5548d8cc-xj2td"
Nov 03 08:40:00.281331 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.281170    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b030cb9d-3c84-4482-9761-a507e84c9204-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c5548d8cc-xj2td\" (UID: \"b030cb9d-3c84-4482-9761-a507e84c9204\") " pod="openshift-monitoring/thanos-querier-c5548d8cc-xj2td"
Nov 03 08:40:00.281331 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.281187    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b030cb9d-3c84-4482-9761-a507e84c9204-metrics-client-ca\") pod \"thanos-querier-c5548d8cc-xj2td\" (UID: \"b030cb9d-3c84-4482-9761-a507e84c9204\") " pod="openshift-monitoring/thanos-querier-c5548d8cc-xj2td"
Nov 03 08:40:00.281331 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.281260    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b030cb9d-3c84-4482-9761-a507e84c9204-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c5548d8cc-xj2td\" (UID: \"b030cb9d-3c84-4482-9761-a507e84c9204\") " pod="openshift-monitoring/thanos-querier-c5548d8cc-xj2td"
Nov 03 08:40:00.281331 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.281303    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6pls\" (UniqueName: \"kubernetes.io/projected/b030cb9d-3c84-4482-9761-a507e84c9204-kube-api-access-d6pls\") pod \"thanos-querier-c5548d8cc-xj2td\" (UID: \"b030cb9d-3c84-4482-9761-a507e84c9204\") " pod="openshift-monitoring/thanos-querier-c5548d8cc-xj2td"
Nov 03 08:40:00.281331 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.281327    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b030cb9d-3c84-4482-9761-a507e84c9204-secret-grpc-tls\") pod \"thanos-querier-c5548d8cc-xj2td\" (UID: \"b030cb9d-3c84-4482-9761-a507e84c9204\") " pod="openshift-monitoring/thanos-querier-c5548d8cc-xj2td"
Nov 03 08:40:00.281567 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.281375    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b030cb9d-3c84-4482-9761-a507e84c9204-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c5548d8cc-xj2td\" (UID: \"b030cb9d-3c84-4482-9761-a507e84c9204\") " pod="openshift-monitoring/thanos-querier-c5548d8cc-xj2td"
Nov 03 08:40:00.281567 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.281426    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b030cb9d-3c84-4482-9761-a507e84c9204-tmp\") pod \"thanos-querier-c5548d8cc-xj2td\" (UID: \"b030cb9d-3c84-4482-9761-a507e84c9204\") " pod="openshift-monitoring/thanos-querier-c5548d8cc-xj2td"
Nov 03 08:40:00.381882 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.381851    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b030cb9d-3c84-4482-9761-a507e84c9204-tmp\") pod \"thanos-querier-c5548d8cc-xj2td\" (UID: \"b030cb9d-3c84-4482-9761-a507e84c9204\") " pod="openshift-monitoring/thanos-querier-c5548d8cc-xj2td"
Nov 03 08:40:00.382106 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.381899    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b030cb9d-3c84-4482-9761-a507e84c9204-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c5548d8cc-xj2td\" (UID: \"b030cb9d-3c84-4482-9761-a507e84c9204\") " pod="openshift-monitoring/thanos-querier-c5548d8cc-xj2td"
Nov 03 08:40:00.382106 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.381920    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b030cb9d-3c84-4482-9761-a507e84c9204-secret-thanos-querier-tls\") pod \"thanos-querier-c5548d8cc-xj2td\" (UID: \"b030cb9d-3c84-4482-9761-a507e84c9204\") " pod="openshift-monitoring/thanos-querier-c5548d8cc-xj2td"
Nov 03 08:40:00.382106 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.381939    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b030cb9d-3c84-4482-9761-a507e84c9204-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c5548d8cc-xj2td\" (UID: \"b030cb9d-3c84-4482-9761-a507e84c9204\") " pod="openshift-monitoring/thanos-querier-c5548d8cc-xj2td"
Nov 03 08:40:00.382106 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.382086    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b030cb9d-3c84-4482-9761-a507e84c9204-metrics-client-ca\") pod \"thanos-querier-c5548d8cc-xj2td\" (UID: \"b030cb9d-3c84-4482-9761-a507e84c9204\") " pod="openshift-monitoring/thanos-querier-c5548d8cc-xj2td"
Nov 03 08:40:00.382313 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.382126    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b030cb9d-3c84-4482-9761-a507e84c9204-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c5548d8cc-xj2td\" (UID: \"b030cb9d-3c84-4482-9761-a507e84c9204\") " pod="openshift-monitoring/thanos-querier-c5548d8cc-xj2td"
Nov 03 08:40:00.382313 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.382156    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6pls\" (UniqueName: \"kubernetes.io/projected/b030cb9d-3c84-4482-9761-a507e84c9204-kube-api-access-d6pls\") pod \"thanos-querier-c5548d8cc-xj2td\" (UID: \"b030cb9d-3c84-4482-9761-a507e84c9204\") " pod="openshift-monitoring/thanos-querier-c5548d8cc-xj2td"
Nov 03 08:40:00.382313 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.382177    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b030cb9d-3c84-4482-9761-a507e84c9204-secret-grpc-tls\") pod \"thanos-querier-c5548d8cc-xj2td\" (UID: \"b030cb9d-3c84-4482-9761-a507e84c9204\") " pod="openshift-monitoring/thanos-querier-c5548d8cc-xj2td"
Nov 03 08:40:00.382313 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.382220    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b030cb9d-3c84-4482-9761-a507e84c9204-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c5548d8cc-xj2td\" (UID: \"b030cb9d-3c84-4482-9761-a507e84c9204\") " pod="openshift-monitoring/thanos-querier-c5548d8cc-xj2td"
Nov 03 08:40:00.382500 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.382379    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b030cb9d-3c84-4482-9761-a507e84c9204-tmp\") pod \"thanos-querier-c5548d8cc-xj2td\" (UID: \"b030cb9d-3c84-4482-9761-a507e84c9204\") " pod="openshift-monitoring/thanos-querier-c5548d8cc-xj2td"
Nov 03 08:40:00.383119 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.383030    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b030cb9d-3c84-4482-9761-a507e84c9204-metrics-client-ca\") pod \"thanos-querier-c5548d8cc-xj2td\" (UID: \"b030cb9d-3c84-4482-9761-a507e84c9204\") " pod="openshift-monitoring/thanos-querier-c5548d8cc-xj2td"
Nov 03 08:40:00.384859 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.384831    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b030cb9d-3c84-4482-9761-a507e84c9204-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c5548d8cc-xj2td\" (UID: \"b030cb9d-3c84-4482-9761-a507e84c9204\") " pod="openshift-monitoring/thanos-querier-c5548d8cc-xj2td"
Nov 03 08:40:00.385249 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.385224    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b030cb9d-3c84-4482-9761-a507e84c9204-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c5548d8cc-xj2td\" (UID: \"b030cb9d-3c84-4482-9761-a507e84c9204\") " pod="openshift-monitoring/thanos-querier-c5548d8cc-xj2td"
Nov 03 08:40:00.385333 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.385245    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b030cb9d-3c84-4482-9761-a507e84c9204-secret-grpc-tls\") pod \"thanos-querier-c5548d8cc-xj2td\" (UID: \"b030cb9d-3c84-4482-9761-a507e84c9204\") " pod="openshift-monitoring/thanos-querier-c5548d8cc-xj2td"
Nov 03 08:40:00.385397 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.385378    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b030cb9d-3c84-4482-9761-a507e84c9204-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c5548d8cc-xj2td\" (UID: \"b030cb9d-3c84-4482-9761-a507e84c9204\") " pod="openshift-monitoring/thanos-querier-c5548d8cc-xj2td"
Nov 03 08:40:00.385436 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.385392    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b030cb9d-3c84-4482-9761-a507e84c9204-secret-thanos-querier-tls\") pod \"thanos-querier-c5548d8cc-xj2td\" (UID: \"b030cb9d-3c84-4482-9761-a507e84c9204\") " pod="openshift-monitoring/thanos-querier-c5548d8cc-xj2td"
Nov 03 08:40:00.385469 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.385453    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b030cb9d-3c84-4482-9761-a507e84c9204-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c5548d8cc-xj2td\" (UID: \"b030cb9d-3c84-4482-9761-a507e84c9204\") " pod="openshift-monitoring/thanos-querier-c5548d8cc-xj2td"
Nov 03 08:40:00.442364 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.442334    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6pls\" (UniqueName: \"kubernetes.io/projected/b030cb9d-3c84-4482-9761-a507e84c9204-kube-api-access-d6pls\") pod \"thanos-querier-c5548d8cc-xj2td\" (UID: \"b030cb9d-3c84-4482-9761-a507e84c9204\") " pod="openshift-monitoring/thanos-querier-c5548d8cc-xj2td"
Nov 03 08:40:00.575863 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.575532    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 08:40:00.578179 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:40:00.578151    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84d30531_a4d7_43e8_b283_4ce98cf06b9d.slice/crio-21cfe31eb73d5a47a14a9072ac52b5198d213f3cd2c6a4e42648bd72734a7759 WatchSource:0}: Error finding container 21cfe31eb73d5a47a14a9072ac52b5198d213f3cd2c6a4e42648bd72734a7759: Status 404 returned error can't find the container with id 21cfe31eb73d5a47a14a9072ac52b5198d213f3cd2c6a4e42648bd72734a7759
Nov 03 08:40:00.593504 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.593483    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-c5548d8cc-xj2td"
Nov 03 08:40:00.995022 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.994920    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hjsfs" event={"ID":"ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9","Type":"ContainerStarted","Data":"eb5d730f9f35c8671b484b0f4d6ebd22037f663c3857882f38852b36efce6388"}
Nov 03 08:40:00.995022 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.995008    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hjsfs" event={"ID":"ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9","Type":"ContainerStarted","Data":"0e99cf6f0f5548b544b16537403cd8d77ed69ae6da141d7ca20dddb553c7f996"}
Nov 03 08:40:00.996193 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:00.996173    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84d30531-a4d7-43e8-b283-4ce98cf06b9d","Type":"ContainerStarted","Data":"21cfe31eb73d5a47a14a9072ac52b5198d213f3cd2c6a4e42648bd72734a7759"}
Nov 03 08:40:01.017380 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:01.017225    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-c5548d8cc-xj2td"]
Nov 03 08:40:01.019776 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:40:01.019744    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb030cb9d_3c84_4482_9761_a507e84c9204.slice/crio-1009c696666c312cf20826a593d0b3c9dff815b3e895c1dfe3cc0f578c7cc23a WatchSource:0}: Error finding container 1009c696666c312cf20826a593d0b3c9dff815b3e895c1dfe3cc0f578c7cc23a: Status 404 returned error can't find the container with id 1009c696666c312cf20826a593d0b3c9dff815b3e895c1dfe3cc0f578c7cc23a
Nov 03 08:40:01.113083 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:01.113023    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hjsfs" podStartSLOduration=2.695906971 podStartE2EDuration="4.113006802s" totalImagesPullingTime="1.342554882s" totalInitContainerRuntime="74.544949ms" isStatefulPod=true podCreationTimestamp="2026-11-03 08:39:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:40:01.112506719 +0000 UTC m=+25.381570153" watchObservedRunningTime="2026-11-03 08:40:01.113006802 +0000 UTC m=+25.382070238"
Nov 03 08:40:02.001307 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.001273    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84d30531-a4d7-43e8-b283-4ce98cf06b9d","Type":"ContainerStarted","Data":"9f492213da003b16ff5cb0a12a1e69c58433d7b14fc186ee12edc1fc3c57245b"}
Nov 03 08:40:02.003598 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.003012    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c5548d8cc-xj2td" event={"ID":"b030cb9d-3c84-4482-9761-a507e84c9204","Type":"ContainerStarted","Data":"1009c696666c312cf20826a593d0b3c9dff815b3e895c1dfe3cc0f578c7cc23a"}
Nov 03 08:40:02.227771 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.227694    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-4dvrv"]
Nov 03 08:40:02.230719 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.230697    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4dvrv"
Nov 03 08:40:02.287527 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.287504    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-m545b\""
Nov 03 08:40:02.287527 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.287519    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 03 08:40:02.292691 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.292659    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-4dvrv"]
Nov 03 08:40:02.299203 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.299181    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/024a7379-f2b5-41ce-95a0-58da9ad664fb-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-4dvrv\" (UID: \"024a7379-f2b5-41ce-95a0-58da9ad664fb\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4dvrv"
Nov 03 08:40:02.299335 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.299235    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzzsz\" (UniqueName: \"kubernetes.io/projected/024a7379-f2b5-41ce-95a0-58da9ad664fb-kube-api-access-kzzsz\") pod \"monitoring-plugin-77f4f76594-4dvrv\" (UID: \"024a7379-f2b5-41ce-95a0-58da9ad664fb\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4dvrv"
Nov 03 08:40:02.399902 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.399871    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/024a7379-f2b5-41ce-95a0-58da9ad664fb-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-4dvrv\" (UID: \"024a7379-f2b5-41ce-95a0-58da9ad664fb\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4dvrv"
Nov 03 08:40:02.400121 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.399971    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzzsz\" (UniqueName: \"kubernetes.io/projected/024a7379-f2b5-41ce-95a0-58da9ad664fb-kube-api-access-kzzsz\") pod \"monitoring-plugin-77f4f76594-4dvrv\" (UID: \"024a7379-f2b5-41ce-95a0-58da9ad664fb\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4dvrv"
Nov 03 08:40:02.403001 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.402973    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/024a7379-f2b5-41ce-95a0-58da9ad664fb-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-4dvrv\" (UID: \"024a7379-f2b5-41ce-95a0-58da9ad664fb\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4dvrv"
Nov 03 08:40:02.460626 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.460595    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzzsz\" (UniqueName: \"kubernetes.io/projected/024a7379-f2b5-41ce-95a0-58da9ad664fb-kube-api-access-kzzsz\") pod \"monitoring-plugin-77f4f76594-4dvrv\" (UID: \"024a7379-f2b5-41ce-95a0-58da9ad664fb\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-4dvrv"
Nov 03 08:40:02.497191 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.497117    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-748b854566-dbwqz"]
Nov 03 08:40:02.500183 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.500162    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-748b854566-dbwqz"
Nov 03 08:40:02.542811 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.542780    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4dvrv"
Nov 03 08:40:02.557883 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.557861    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 08:40:02.558016 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.557931    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 03 08:40:02.558016 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.557931    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 03 08:40:02.558016 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.558014    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-hfq6d\""
Nov 03 08:40:02.558145 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.558087    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 03 08:40:02.564469 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.564449    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-748b854566-dbwqz"]
Nov 03 08:40:02.603470 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.601766    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/809f0f9a-ccb4-4bca-b6dc-1af7b9a66555-client-ca-bundle\") pod \"metrics-server-748b854566-dbwqz\" (UID: \"809f0f9a-ccb4-4bca-b6dc-1af7b9a66555\") " pod="openshift-monitoring/metrics-server-748b854566-dbwqz"
Nov 03 08:40:02.603470 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.601820    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxq8s\" (UniqueName: \"kubernetes.io/projected/809f0f9a-ccb4-4bca-b6dc-1af7b9a66555-kube-api-access-nxq8s\") pod \"metrics-server-748b854566-dbwqz\" (UID: \"809f0f9a-ccb4-4bca-b6dc-1af7b9a66555\") " pod="openshift-monitoring/metrics-server-748b854566-dbwqz"
Nov 03 08:40:02.603470 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.601854    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/809f0f9a-ccb4-4bca-b6dc-1af7b9a66555-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-748b854566-dbwqz\" (UID: \"809f0f9a-ccb4-4bca-b6dc-1af7b9a66555\") " pod="openshift-monitoring/metrics-server-748b854566-dbwqz"
Nov 03 08:40:02.603470 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.602046    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/809f0f9a-ccb4-4bca-b6dc-1af7b9a66555-audit-log\") pod \"metrics-server-748b854566-dbwqz\" (UID: \"809f0f9a-ccb4-4bca-b6dc-1af7b9a66555\") " pod="openshift-monitoring/metrics-server-748b854566-dbwqz"
Nov 03 08:40:02.603470 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.602233    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/809f0f9a-ccb4-4bca-b6dc-1af7b9a66555-secret-metrics-server-client-certs\") pod \"metrics-server-748b854566-dbwqz\" (UID: \"809f0f9a-ccb4-4bca-b6dc-1af7b9a66555\") " pod="openshift-monitoring/metrics-server-748b854566-dbwqz"
Nov 03 08:40:02.603470 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.602350    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/809f0f9a-ccb4-4bca-b6dc-1af7b9a66555-secret-metrics-server-tls\") pod \"metrics-server-748b854566-dbwqz\" (UID: \"809f0f9a-ccb4-4bca-b6dc-1af7b9a66555\") " pod="openshift-monitoring/metrics-server-748b854566-dbwqz"
Nov 03 08:40:02.603470 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.602390    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/809f0f9a-ccb4-4bca-b6dc-1af7b9a66555-metrics-server-audit-profiles\") pod \"metrics-server-748b854566-dbwqz\" (UID: \"809f0f9a-ccb4-4bca-b6dc-1af7b9a66555\") " pod="openshift-monitoring/metrics-server-748b854566-dbwqz"
Nov 03 08:40:02.658351 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.658312    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dp4pbof07np48\""
Nov 03 08:40:02.703817 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.703786    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/809f0f9a-ccb4-4bca-b6dc-1af7b9a66555-secret-metrics-server-tls\") pod \"metrics-server-748b854566-dbwqz\" (UID: \"809f0f9a-ccb4-4bca-b6dc-1af7b9a66555\") " pod="openshift-monitoring/metrics-server-748b854566-dbwqz"
Nov 03 08:40:02.704013 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.703824    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/809f0f9a-ccb4-4bca-b6dc-1af7b9a66555-metrics-server-audit-profiles\") pod \"metrics-server-748b854566-dbwqz\" (UID: \"809f0f9a-ccb4-4bca-b6dc-1af7b9a66555\") " pod="openshift-monitoring/metrics-server-748b854566-dbwqz"
Nov 03 08:40:02.704013 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.703850    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/809f0f9a-ccb4-4bca-b6dc-1af7b9a66555-client-ca-bundle\") pod \"metrics-server-748b854566-dbwqz\" (UID: \"809f0f9a-ccb4-4bca-b6dc-1af7b9a66555\") " pod="openshift-monitoring/metrics-server-748b854566-dbwqz"
Nov 03 08:40:02.704013 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.703878    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxq8s\" (UniqueName: \"kubernetes.io/projected/809f0f9a-ccb4-4bca-b6dc-1af7b9a66555-kube-api-access-nxq8s\") pod \"metrics-server-748b854566-dbwqz\" (UID: \"809f0f9a-ccb4-4bca-b6dc-1af7b9a66555\") " pod="openshift-monitoring/metrics-server-748b854566-dbwqz"
Nov 03 08:40:02.704013 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.703898    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/809f0f9a-ccb4-4bca-b6dc-1af7b9a66555-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-748b854566-dbwqz\" (UID: \"809f0f9a-ccb4-4bca-b6dc-1af7b9a66555\") " pod="openshift-monitoring/metrics-server-748b854566-dbwqz"
Nov 03 08:40:02.704013 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.703921    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/809f0f9a-ccb4-4bca-b6dc-1af7b9a66555-audit-log\") pod \"metrics-server-748b854566-dbwqz\" (UID: \"809f0f9a-ccb4-4bca-b6dc-1af7b9a66555\") " pod="openshift-monitoring/metrics-server-748b854566-dbwqz"
Nov 03 08:40:02.704013 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.704002    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/809f0f9a-ccb4-4bca-b6dc-1af7b9a66555-secret-metrics-server-client-certs\") pod \"metrics-server-748b854566-dbwqz\" (UID: \"809f0f9a-ccb4-4bca-b6dc-1af7b9a66555\") " pod="openshift-monitoring/metrics-server-748b854566-dbwqz"
Nov 03 08:40:02.704871 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.704846    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/809f0f9a-ccb4-4bca-b6dc-1af7b9a66555-audit-log\") pod \"metrics-server-748b854566-dbwqz\" (UID: \"809f0f9a-ccb4-4bca-b6dc-1af7b9a66555\") " pod="openshift-monitoring/metrics-server-748b854566-dbwqz"
Nov 03 08:40:02.705478 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.705450    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/809f0f9a-ccb4-4bca-b6dc-1af7b9a66555-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-748b854566-dbwqz\" (UID: \"809f0f9a-ccb4-4bca-b6dc-1af7b9a66555\") " pod="openshift-monitoring/metrics-server-748b854566-dbwqz"
Nov 03 08:40:02.706787 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.706754    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/809f0f9a-ccb4-4bca-b6dc-1af7b9a66555-metrics-server-audit-profiles\") pod \"metrics-server-748b854566-dbwqz\" (UID: \"809f0f9a-ccb4-4bca-b6dc-1af7b9a66555\") " pod="openshift-monitoring/metrics-server-748b854566-dbwqz"
Nov 03 08:40:02.706968 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.706923    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/809f0f9a-ccb4-4bca-b6dc-1af7b9a66555-secret-metrics-server-client-certs\") pod \"metrics-server-748b854566-dbwqz\" (UID: \"809f0f9a-ccb4-4bca-b6dc-1af7b9a66555\") " pod="openshift-monitoring/metrics-server-748b854566-dbwqz"
Nov 03 08:40:02.707337 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.707316    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/809f0f9a-ccb4-4bca-b6dc-1af7b9a66555-client-ca-bundle\") pod \"metrics-server-748b854566-dbwqz\" (UID: \"809f0f9a-ccb4-4bca-b6dc-1af7b9a66555\") " pod="openshift-monitoring/metrics-server-748b854566-dbwqz"
Nov 03 08:40:02.708354 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.708334    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/809f0f9a-ccb4-4bca-b6dc-1af7b9a66555-secret-metrics-server-tls\") pod \"metrics-server-748b854566-dbwqz\" (UID: \"809f0f9a-ccb4-4bca-b6dc-1af7b9a66555\") " pod="openshift-monitoring/metrics-server-748b854566-dbwqz"
Nov 03 08:40:02.763814 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.763739    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxq8s\" (UniqueName: \"kubernetes.io/projected/809f0f9a-ccb4-4bca-b6dc-1af7b9a66555-kube-api-access-nxq8s\") pod \"metrics-server-748b854566-dbwqz\" (UID: \"809f0f9a-ccb4-4bca-b6dc-1af7b9a66555\") " pod="openshift-monitoring/metrics-server-748b854566-dbwqz"
Nov 03 08:40:02.814136 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.813922    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-748b854566-dbwqz"
Nov 03 08:40:02.925768 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:02.925737    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-4dvrv"]
Nov 03 08:40:02.928808 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:40:02.928776    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod024a7379_f2b5_41ce_95a0_58da9ad664fb.slice/crio-141785941d5fd0093b3c2f0d6096b2e9720ed3b43b71a12e71eca10331d42f33 WatchSource:0}: Error finding container 141785941d5fd0093b3c2f0d6096b2e9720ed3b43b71a12e71eca10331d42f33: Status 404 returned error can't find the container with id 141785941d5fd0093b3c2f0d6096b2e9720ed3b43b71a12e71eca10331d42f33
Nov 03 08:40:03.008546 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:03.008512    4473 generic.go:356] "Generic (PLEG): container finished" podID="84d30531-a4d7-43e8-b283-4ce98cf06b9d" containerID="9f492213da003b16ff5cb0a12a1e69c58433d7b14fc186ee12edc1fc3c57245b" exitCode=0
Nov 03 08:40:03.009063 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:03.008599    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84d30531-a4d7-43e8-b283-4ce98cf06b9d","Type":"ContainerDied","Data":"9f492213da003b16ff5cb0a12a1e69c58433d7b14fc186ee12edc1fc3c57245b"}
Nov 03 08:40:03.010360 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:03.010322    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4dvrv" event={"ID":"024a7379-f2b5-41ce-95a0-58da9ad664fb","Type":"ContainerStarted","Data":"141785941d5fd0093b3c2f0d6096b2e9720ed3b43b71a12e71eca10331d42f33"}
Nov 03 08:40:03.190959 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:03.190924    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-748b854566-dbwqz"]
Nov 03 08:40:03.193855 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:40:03.193826    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod809f0f9a_ccb4_4bca_b6dc_1af7b9a66555.slice/crio-a825dd24eacbf2e4b9e6542c3fc7366a96eb46bf001bb58a423471a1d178d588 WatchSource:0}: Error finding container a825dd24eacbf2e4b9e6542c3fc7366a96eb46bf001bb58a423471a1d178d588: Status 404 returned error can't find the container with id a825dd24eacbf2e4b9e6542c3fc7366a96eb46bf001bb58a423471a1d178d588
Nov 03 08:40:04.017677 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:04.017639    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c5548d8cc-xj2td" event={"ID":"b030cb9d-3c84-4482-9761-a507e84c9204","Type":"ContainerStarted","Data":"9542dcc542fbaedba1d103a3ab60a19b2a3aeba1cd8648475039694875605923"}
Nov 03 08:40:04.018159 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:04.017727    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c5548d8cc-xj2td" event={"ID":"b030cb9d-3c84-4482-9761-a507e84c9204","Type":"ContainerStarted","Data":"f40203324c6dba8835c164a6819917667380ced15fe8328671dfda255ad2aef1"}
Nov 03 08:40:04.018159 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:04.017768    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c5548d8cc-xj2td" event={"ID":"b030cb9d-3c84-4482-9761-a507e84c9204","Type":"ContainerStarted","Data":"cbdae89a054bab85fd40be2d134c411ab47764a8eb9cb8b7102b088ca3335116"}
Nov 03 08:40:04.019451 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:04.019425    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-748b854566-dbwqz" event={"ID":"809f0f9a-ccb4-4bca-b6dc-1af7b9a66555","Type":"ContainerStarted","Data":"a825dd24eacbf2e4b9e6542c3fc7366a96eb46bf001bb58a423471a1d178d588"}
Nov 03 08:40:05.039271 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:05.039071    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c5548d8cc-xj2td" event={"ID":"b030cb9d-3c84-4482-9761-a507e84c9204","Type":"ContainerStarted","Data":"fb2fb2701591be6b81337026d14e068cd211e704872b20115784af22343addf0"}
Nov 03 08:40:05.039271 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:05.039197    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c5548d8cc-xj2td" event={"ID":"b030cb9d-3c84-4482-9761-a507e84c9204","Type":"ContainerStarted","Data":"99b4c16d8ab8a6121bfb4d723cba6d0550787550eb6aa3bcd113a468c8f69be2"}
Nov 03 08:40:06.045911 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:06.045868    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c5548d8cc-xj2td" event={"ID":"b030cb9d-3c84-4482-9761-a507e84c9204","Type":"ContainerStarted","Data":"823e01ec7b9d00e8199529a29a8b2336e06eeca8431a55ff91906b33d74d2185"}
Nov 03 08:40:06.046392 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:06.046092    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-c5548d8cc-xj2td"
Nov 03 08:40:06.047516 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:06.047491    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-748b854566-dbwqz" event={"ID":"809f0f9a-ccb4-4bca-b6dc-1af7b9a66555","Type":"ContainerStarted","Data":"facccfae2d624f1ac4abdf8cf804abe5a7464cf7b67b94f54029efa0c12c2c0d"}
Nov 03 08:40:06.050772 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:06.050744    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84d30531-a4d7-43e8-b283-4ce98cf06b9d","Type":"ContainerStarted","Data":"23269a172b5bec18a9d604863f59344b206b1b205ca2239130d3d645d9e1219b"}
Nov 03 08:40:06.050868 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:06.050817    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84d30531-a4d7-43e8-b283-4ce98cf06b9d","Type":"ContainerStarted","Data":"62d7f13ce9b644220d38ca20d41268b0d2f8b547669da38d7bbeb7057bb69474"}
Nov 03 08:40:06.050868 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:06.050862    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84d30531-a4d7-43e8-b283-4ce98cf06b9d","Type":"ContainerStarted","Data":"de39a77d1b8bddb52de41b7e92d3418b8bd98a6ad68483d7747fd07dfefa74df"}
Nov 03 08:40:06.051017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:06.050901    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84d30531-a4d7-43e8-b283-4ce98cf06b9d","Type":"ContainerStarted","Data":"b53622a4893ef5a261353f2fbe6c9950fbb88ef5cd8c7ca83ba9221aa83e90ad"}
Nov 03 08:40:06.051017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:06.050938    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84d30531-a4d7-43e8-b283-4ce98cf06b9d","Type":"ContainerStarted","Data":"acd60427a52136e3bbeabda0fa0773303f0b0f058f86350638af07947c718f75"}
Nov 03 08:40:06.051017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:06.050999    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84d30531-a4d7-43e8-b283-4ce98cf06b9d","Type":"ContainerStarted","Data":"87bedfa3971e96aa58a2edccd8bc6889eb5928eb6bb363486ca2653860354c2b"}
Nov 03 08:40:06.052192 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:06.052171    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4dvrv" event={"ID":"024a7379-f2b5-41ce-95a0-58da9ad664fb","Type":"ContainerStarted","Data":"49c6a38ece3a5267df711ff8798dff535210b0061596bcebe87fc7530c076eaa"}
Nov 03 08:40:06.052406 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:06.052387    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4dvrv"
Nov 03 08:40:06.057506 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:06.057490    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4dvrv"
Nov 03 08:40:06.354397 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:06.354300    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.766537989 podStartE2EDuration="7.354286595s" totalImagesPullingTime="3.563902685s" totalInitContainerRuntime="23.845921ms" isStatefulPod=false podCreationTimestamp="2026-11-03 08:39:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:40:06.353371083 +0000 UTC m=+30.622434528" watchObservedRunningTime="2026-11-03 08:40:06.354286595 +0000 UTC m=+30.623350030"
Nov 03 08:40:06.354704 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:06.354683    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-c5548d8cc-xj2td" podStartSLOduration=2.7031823360000002 podStartE2EDuration="6.354678264s" totalImagesPullingTime="3.651495928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:40:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:40:06.227372365 +0000 UTC m=+30.496435799" watchObservedRunningTime="2026-11-03 08:40:06.354678264 +0000 UTC m=+30.623741725"
Nov 03 08:40:06.472146 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:06.472096    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-748b854566-dbwqz" podStartSLOduration=2.454031928 podStartE2EDuration="4.472081489s" totalImagesPullingTime="2.018049561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:40:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:40:06.471012112 +0000 UTC m=+30.740075562" watchObservedRunningTime="2026-11-03 08:40:06.472081489 +0000 UTC m=+30.741144901"
Nov 03 08:40:06.489099 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:06.489075    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-clx84"
Nov 03 08:40:06.586768 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:06.586716    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-4dvrv" podStartSLOduration=2.499272103 podStartE2EDuration="4.586701626s" totalImagesPullingTime="2.087429523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:40:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:40:06.58540613 +0000 UTC m=+30.854469584" watchObservedRunningTime="2026-11-03 08:40:06.586701626 +0000 UTC m=+30.855765060"
Nov 03 08:40:12.064811 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:12.064784    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-c5548d8cc-xj2td"
Nov 03 08:40:16.935602 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:40:16.935525    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:40:17.809078 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:17.809051    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-65dff66875-qcbkf"]
Nov 03 08:40:17.810970 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:17.810928    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65dff66875-qcbkf"
Nov 03 08:40:17.917377 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:17.917257    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 08:40:17.917377 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:17.917305    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 08:40:17.917377 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:17.917370    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 08:40:17.917377 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:17.917310    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bbkmr\""
Nov 03 08:40:17.917377 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:17.917381    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 08:40:17.917715 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:17.917349    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 08:40:17.917715 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:17.917440    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 08:40:17.917715 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:17.917447    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 08:40:17.922831 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:17.922809    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65dff66875-qcbkf"]
Nov 03 08:40:17.927919 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:17.927899    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa0e821d-8b15-414d-b62e-7a311590e1d2-service-ca\") pod \"console-65dff66875-qcbkf\" (UID: \"aa0e821d-8b15-414d-b62e-7a311590e1d2\") " pod="openshift-console/console-65dff66875-qcbkf"
Nov 03 08:40:17.928033 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:17.927927    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw87b\" (UniqueName: \"kubernetes.io/projected/aa0e821d-8b15-414d-b62e-7a311590e1d2-kube-api-access-fw87b\") pod \"console-65dff66875-qcbkf\" (UID: \"aa0e821d-8b15-414d-b62e-7a311590e1d2\") " pod="openshift-console/console-65dff66875-qcbkf"
Nov 03 08:40:17.928073 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:17.928024    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa0e821d-8b15-414d-b62e-7a311590e1d2-console-config\") pod \"console-65dff66875-qcbkf\" (UID: \"aa0e821d-8b15-414d-b62e-7a311590e1d2\") " pod="openshift-console/console-65dff66875-qcbkf"
Nov 03 08:40:17.928073 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:17.928064    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa0e821d-8b15-414d-b62e-7a311590e1d2-console-oauth-config\") pod \"console-65dff66875-qcbkf\" (UID: \"aa0e821d-8b15-414d-b62e-7a311590e1d2\") " pod="openshift-console/console-65dff66875-qcbkf"
Nov 03 08:40:17.928158 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:17.928086    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa0e821d-8b15-414d-b62e-7a311590e1d2-oauth-serving-cert\") pod \"console-65dff66875-qcbkf\" (UID: \"aa0e821d-8b15-414d-b62e-7a311590e1d2\") " pod="openshift-console/console-65dff66875-qcbkf"
Nov 03 08:40:17.928158 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:17.928125    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa0e821d-8b15-414d-b62e-7a311590e1d2-console-serving-cert\") pod \"console-65dff66875-qcbkf\" (UID: \"aa0e821d-8b15-414d-b62e-7a311590e1d2\") " pod="openshift-console/console-65dff66875-qcbkf"
Nov 03 08:40:17.928229 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:17.928182    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa0e821d-8b15-414d-b62e-7a311590e1d2-tmp\") pod \"console-65dff66875-qcbkf\" (UID: \"aa0e821d-8b15-414d-b62e-7a311590e1d2\") " pod="openshift-console/console-65dff66875-qcbkf"
Nov 03 08:40:18.029054 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:18.029021    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa0e821d-8b15-414d-b62e-7a311590e1d2-console-config\") pod \"console-65dff66875-qcbkf\" (UID: \"aa0e821d-8b15-414d-b62e-7a311590e1d2\") " pod="openshift-console/console-65dff66875-qcbkf"
Nov 03 08:40:18.029473 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:18.029064    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa0e821d-8b15-414d-b62e-7a311590e1d2-console-oauth-config\") pod \"console-65dff66875-qcbkf\" (UID: \"aa0e821d-8b15-414d-b62e-7a311590e1d2\") " pod="openshift-console/console-65dff66875-qcbkf"
Nov 03 08:40:18.029473 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:18.029089    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa0e821d-8b15-414d-b62e-7a311590e1d2-oauth-serving-cert\") pod \"console-65dff66875-qcbkf\" (UID: \"aa0e821d-8b15-414d-b62e-7a311590e1d2\") " pod="openshift-console/console-65dff66875-qcbkf"
Nov 03 08:40:18.029473 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:18.029111    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa0e821d-8b15-414d-b62e-7a311590e1d2-console-serving-cert\") pod \"console-65dff66875-qcbkf\" (UID: \"aa0e821d-8b15-414d-b62e-7a311590e1d2\") " pod="openshift-console/console-65dff66875-qcbkf"
Nov 03 08:40:18.029473 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:18.029162    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa0e821d-8b15-414d-b62e-7a311590e1d2-tmp\") pod \"console-65dff66875-qcbkf\" (UID: \"aa0e821d-8b15-414d-b62e-7a311590e1d2\") " pod="openshift-console/console-65dff66875-qcbkf"
Nov 03 08:40:18.029473 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:18.029218    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa0e821d-8b15-414d-b62e-7a311590e1d2-service-ca\") pod \"console-65dff66875-qcbkf\" (UID: \"aa0e821d-8b15-414d-b62e-7a311590e1d2\") " pod="openshift-console/console-65dff66875-qcbkf"
Nov 03 08:40:18.029473 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:18.029245    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw87b\" (UniqueName: \"kubernetes.io/projected/aa0e821d-8b15-414d-b62e-7a311590e1d2-kube-api-access-fw87b\") pod \"console-65dff66875-qcbkf\" (UID: \"aa0e821d-8b15-414d-b62e-7a311590e1d2\") " pod="openshift-console/console-65dff66875-qcbkf"
Nov 03 08:40:18.029714 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:18.029684    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa0e821d-8b15-414d-b62e-7a311590e1d2-tmp\") pod \"console-65dff66875-qcbkf\" (UID: \"aa0e821d-8b15-414d-b62e-7a311590e1d2\") " pod="openshift-console/console-65dff66875-qcbkf"
Nov 03 08:40:18.029920 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:18.029904    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa0e821d-8b15-414d-b62e-7a311590e1d2-service-ca\") pod \"console-65dff66875-qcbkf\" (UID: \"aa0e821d-8b15-414d-b62e-7a311590e1d2\") " pod="openshift-console/console-65dff66875-qcbkf"
Nov 03 08:40:18.030061 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:18.030042    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa0e821d-8b15-414d-b62e-7a311590e1d2-console-config\") pod \"console-65dff66875-qcbkf\" (UID: \"aa0e821d-8b15-414d-b62e-7a311590e1d2\") " pod="openshift-console/console-65dff66875-qcbkf"
Nov 03 08:40:18.030162 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:18.030140    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa0e821d-8b15-414d-b62e-7a311590e1d2-oauth-serving-cert\") pod \"console-65dff66875-qcbkf\" (UID: \"aa0e821d-8b15-414d-b62e-7a311590e1d2\") " pod="openshift-console/console-65dff66875-qcbkf"
Nov 03 08:40:18.031669 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:18.031651    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa0e821d-8b15-414d-b62e-7a311590e1d2-console-oauth-config\") pod \"console-65dff66875-qcbkf\" (UID: \"aa0e821d-8b15-414d-b62e-7a311590e1d2\") " pod="openshift-console/console-65dff66875-qcbkf"
Nov 03 08:40:18.031714 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:18.031660    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa0e821d-8b15-414d-b62e-7a311590e1d2-console-serving-cert\") pod \"console-65dff66875-qcbkf\" (UID: \"aa0e821d-8b15-414d-b62e-7a311590e1d2\") " pod="openshift-console/console-65dff66875-qcbkf"
Nov 03 08:40:18.089098 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:18.089048    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw87b\" (UniqueName: \"kubernetes.io/projected/aa0e821d-8b15-414d-b62e-7a311590e1d2-kube-api-access-fw87b\") pod \"console-65dff66875-qcbkf\" (UID: \"aa0e821d-8b15-414d-b62e-7a311590e1d2\") " pod="openshift-console/console-65dff66875-qcbkf"
Nov 03 08:40:18.120805 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:18.120780    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65dff66875-qcbkf"
Nov 03 08:40:18.492764 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:18.492741    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65dff66875-qcbkf"]
Nov 03 08:40:18.495284 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:40:18.495261    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa0e821d_8b15_414d_b62e_7a311590e1d2.slice/crio-aa1784b440a7fc755075d35d30844daf0a2caf41e0365605dfc6bfbe06e5a34e WatchSource:0}: Error finding container aa1784b440a7fc755075d35d30844daf0a2caf41e0365605dfc6bfbe06e5a34e: Status 404 returned error can't find the container with id aa1784b440a7fc755075d35d30844daf0a2caf41e0365605dfc6bfbe06e5a34e
Nov 03 08:40:19.105151 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:19.105114    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65dff66875-qcbkf" event={"ID":"aa0e821d-8b15-414d-b62e-7a311590e1d2","Type":"ContainerStarted","Data":"aa1784b440a7fc755075d35d30844daf0a2caf41e0365605dfc6bfbe06e5a34e"}
Nov 03 08:40:22.117170 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:22.117135    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65dff66875-qcbkf" event={"ID":"aa0e821d-8b15-414d-b62e-7a311590e1d2","Type":"ContainerStarted","Data":"f4057991ee1ace24cd2a39e44530fee09ca3333c27dc74244ea242aa607cf231"}
Nov 03 08:40:22.261446 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:22.261395    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-65dff66875-qcbkf" podStartSLOduration=2.238493668 podStartE2EDuration="5.261381167s" totalImagesPullingTime="3.022887499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:40:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:40:22.260285844 +0000 UTC m=+46.529349295" watchObservedRunningTime="2026-11-03 08:40:22.261381167 +0000 UTC m=+46.530444674"
Nov 03 08:40:22.814409 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:22.814377    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-748b854566-dbwqz"
Nov 03 08:40:22.814572 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:22.814519    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-748b854566-dbwqz"
Nov 03 08:40:28.121353 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:28.121315    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65dff66875-qcbkf"
Nov 03 08:40:28.121840 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:28.121372    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65dff66875-qcbkf"
Nov 03 08:40:28.126165 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:28.126147    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65dff66875-qcbkf"
Nov 03 08:40:28.144624 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:28.144608    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65dff66875-qcbkf"
Nov 03 08:40:28.693468 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:28.693436    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c9db68cb9-qq24h"]
Nov 03 08:40:28.695889 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:28.695873    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c9db68cb9-qq24h"
Nov 03 08:40:28.817797 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:28.817769    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bb543947-8e68-4d9e-a352-f546e5249358-console-config\") pod \"console-6c9db68cb9-qq24h\" (UID: \"bb543947-8e68-4d9e-a352-f546e5249358\") " pod="openshift-console/console-6c9db68cb9-qq24h"
Nov 03 08:40:28.817797 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:28.817796    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bb543947-8e68-4d9e-a352-f546e5249358-console-serving-cert\") pod \"console-6c9db68cb9-qq24h\" (UID: \"bb543947-8e68-4d9e-a352-f546e5249358\") " pod="openshift-console/console-6c9db68cb9-qq24h"
Nov 03 08:40:28.818043 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:28.817817    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bb543947-8e68-4d9e-a352-f546e5249358-console-oauth-config\") pod \"console-6c9db68cb9-qq24h\" (UID: \"bb543947-8e68-4d9e-a352-f546e5249358\") " pod="openshift-console/console-6c9db68cb9-qq24h"
Nov 03 08:40:28.818043 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:28.817837    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bb543947-8e68-4d9e-a352-f546e5249358-oauth-serving-cert\") pod \"console-6c9db68cb9-qq24h\" (UID: \"bb543947-8e68-4d9e-a352-f546e5249358\") " pod="openshift-console/console-6c9db68cb9-qq24h"
Nov 03 08:40:28.818043 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:28.817971    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb543947-8e68-4d9e-a352-f546e5249358-tmp\") pod \"console-6c9db68cb9-qq24h\" (UID: \"bb543947-8e68-4d9e-a352-f546e5249358\") " pod="openshift-console/console-6c9db68cb9-qq24h"
Nov 03 08:40:28.818147 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:28.818044    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bb543947-8e68-4d9e-a352-f546e5249358-service-ca\") pod \"console-6c9db68cb9-qq24h\" (UID: \"bb543947-8e68-4d9e-a352-f546e5249358\") " pod="openshift-console/console-6c9db68cb9-qq24h"
Nov 03 08:40:28.818147 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:28.818076    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mvwv\" (UniqueName: \"kubernetes.io/projected/bb543947-8e68-4d9e-a352-f546e5249358-kube-api-access-2mvwv\") pod \"console-6c9db68cb9-qq24h\" (UID: \"bb543947-8e68-4d9e-a352-f546e5249358\") " pod="openshift-console/console-6c9db68cb9-qq24h"
Nov 03 08:40:28.818147 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:28.818095    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb543947-8e68-4d9e-a352-f546e5249358-trusted-ca-bundle\") pod \"console-6c9db68cb9-qq24h\" (UID: \"bb543947-8e68-4d9e-a352-f546e5249358\") " pod="openshift-console/console-6c9db68cb9-qq24h"
Nov 03 08:40:28.901282 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:28.901249    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 03 08:40:28.902259 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:28.902241    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c9db68cb9-qq24h"]
Nov 03 08:40:28.918532 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:28.918510    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bb543947-8e68-4d9e-a352-f546e5249358-console-oauth-config\") pod \"console-6c9db68cb9-qq24h\" (UID: \"bb543947-8e68-4d9e-a352-f546e5249358\") " pod="openshift-console/console-6c9db68cb9-qq24h"
Nov 03 08:40:28.918608 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:28.918537    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bb543947-8e68-4d9e-a352-f546e5249358-oauth-serving-cert\") pod \"console-6c9db68cb9-qq24h\" (UID: \"bb543947-8e68-4d9e-a352-f546e5249358\") " pod="openshift-console/console-6c9db68cb9-qq24h"
Nov 03 08:40:28.918608 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:28.918552    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb543947-8e68-4d9e-a352-f546e5249358-tmp\") pod \"console-6c9db68cb9-qq24h\" (UID: \"bb543947-8e68-4d9e-a352-f546e5249358\") " pod="openshift-console/console-6c9db68cb9-qq24h"
Nov 03 08:40:28.918771 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:28.918634    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bb543947-8e68-4d9e-a352-f546e5249358-service-ca\") pod \"console-6c9db68cb9-qq24h\" (UID: \"bb543947-8e68-4d9e-a352-f546e5249358\") " pod="openshift-console/console-6c9db68cb9-qq24h"
Nov 03 08:40:28.918771 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:28.918696    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mvwv\" (UniqueName: \"kubernetes.io/projected/bb543947-8e68-4d9e-a352-f546e5249358-kube-api-access-2mvwv\") pod \"console-6c9db68cb9-qq24h\" (UID: \"bb543947-8e68-4d9e-a352-f546e5249358\") " pod="openshift-console/console-6c9db68cb9-qq24h"
Nov 03 08:40:28.918771 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:28.918729    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb543947-8e68-4d9e-a352-f546e5249358-trusted-ca-bundle\") pod \"console-6c9db68cb9-qq24h\" (UID: \"bb543947-8e68-4d9e-a352-f546e5249358\") " pod="openshift-console/console-6c9db68cb9-qq24h"
Nov 03 08:40:28.918920 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:28.918774    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bb543947-8e68-4d9e-a352-f546e5249358-console-config\") pod \"console-6c9db68cb9-qq24h\" (UID: \"bb543947-8e68-4d9e-a352-f546e5249358\") " pod="openshift-console/console-6c9db68cb9-qq24h"
Nov 03 08:40:28.918920 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:28.918800    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bb543947-8e68-4d9e-a352-f546e5249358-console-serving-cert\") pod \"console-6c9db68cb9-qq24h\" (UID: \"bb543947-8e68-4d9e-a352-f546e5249358\") " pod="openshift-console/console-6c9db68cb9-qq24h"
Nov 03 08:40:28.918920 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:28.918903    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb543947-8e68-4d9e-a352-f546e5249358-tmp\") pod \"console-6c9db68cb9-qq24h\" (UID: \"bb543947-8e68-4d9e-a352-f546e5249358\") " pod="openshift-console/console-6c9db68cb9-qq24h"
Nov 03 08:40:28.919336 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:28.919311    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bb543947-8e68-4d9e-a352-f546e5249358-oauth-serving-cert\") pod \"console-6c9db68cb9-qq24h\" (UID: \"bb543947-8e68-4d9e-a352-f546e5249358\") " pod="openshift-console/console-6c9db68cb9-qq24h"
Nov 03 08:40:28.919465 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:28.919445    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bb543947-8e68-4d9e-a352-f546e5249358-service-ca\") pod \"console-6c9db68cb9-qq24h\" (UID: \"bb543947-8e68-4d9e-a352-f546e5249358\") " pod="openshift-console/console-6c9db68cb9-qq24h"
Nov 03 08:40:28.919522 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:28.919469    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bb543947-8e68-4d9e-a352-f546e5249358-console-config\") pod \"console-6c9db68cb9-qq24h\" (UID: \"bb543947-8e68-4d9e-a352-f546e5249358\") " pod="openshift-console/console-6c9db68cb9-qq24h"
Nov 03 08:40:28.920316 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:28.920294    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb543947-8e68-4d9e-a352-f546e5249358-trusted-ca-bundle\") pod \"console-6c9db68cb9-qq24h\" (UID: \"bb543947-8e68-4d9e-a352-f546e5249358\") " pod="openshift-console/console-6c9db68cb9-qq24h"
Nov 03 08:40:28.921398 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:28.921366    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bb543947-8e68-4d9e-a352-f546e5249358-console-oauth-config\") pod \"console-6c9db68cb9-qq24h\" (UID: \"bb543947-8e68-4d9e-a352-f546e5249358\") " pod="openshift-console/console-6c9db68cb9-qq24h"
Nov 03 08:40:28.921494 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:28.921452    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bb543947-8e68-4d9e-a352-f546e5249358-console-serving-cert\") pod \"console-6c9db68cb9-qq24h\" (UID: \"bb543947-8e68-4d9e-a352-f546e5249358\") " pod="openshift-console/console-6c9db68cb9-qq24h"
Nov 03 08:40:28.978555 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:28.978499    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mvwv\" (UniqueName: \"kubernetes.io/projected/bb543947-8e68-4d9e-a352-f546e5249358-kube-api-access-2mvwv\") pod \"console-6c9db68cb9-qq24h\" (UID: \"bb543947-8e68-4d9e-a352-f546e5249358\") " pod="openshift-console/console-6c9db68cb9-qq24h"
Nov 03 08:40:29.005646 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:29.005627    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c9db68cb9-qq24h"
Nov 03 08:40:29.382613 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:29.382584    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c9db68cb9-qq24h"]
Nov 03 08:40:29.389211 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:40:29.389184    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb543947_8e68_4d9e_a352_f546e5249358.slice/crio-22f8fd714ff6f7bb22586e5669358e1b0ee3d18bc5ba2e6b291aea529551dbba WatchSource:0}: Error finding container 22f8fd714ff6f7bb22586e5669358e1b0ee3d18bc5ba2e6b291aea529551dbba: Status 404 returned error can't find the container with id 22f8fd714ff6f7bb22586e5669358e1b0ee3d18bc5ba2e6b291aea529551dbba
Nov 03 08:40:30.153696 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:30.153656    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c9db68cb9-qq24h" event={"ID":"bb543947-8e68-4d9e-a352-f546e5249358","Type":"ContainerStarted","Data":"d3cf451241e268198a7009022f6025c7b913be09e6976844bd6e216879fddf8d"}
Nov 03 08:40:30.153860 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:30.153745    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c9db68cb9-qq24h" event={"ID":"bb543947-8e68-4d9e-a352-f546e5249358","Type":"ContainerStarted","Data":"22f8fd714ff6f7bb22586e5669358e1b0ee3d18bc5ba2e6b291aea529551dbba"}
Nov 03 08:40:30.273921 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:30.273877    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6c9db68cb9-qq24h" podStartSLOduration=2.273865645 podStartE2EDuration="2.273865645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:40:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:40:30.272817793 +0000 UTC m=+54.541881252" watchObservedRunningTime="2026-11-03 08:40:30.273865645 +0000 UTC m=+54.542929079"
Nov 03 08:40:39.006000 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:39.005969    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c9db68cb9-qq24h"
Nov 03 08:40:39.006371 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:39.006022    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c9db68cb9-qq24h"
Nov 03 08:40:39.010535 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:39.010514    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c9db68cb9-qq24h"
Nov 03 08:40:39.188019 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:39.187989    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c9db68cb9-qq24h"
Nov 03 08:40:39.334201 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:39.334126    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65dff66875-qcbkf"]
Nov 03 08:40:42.819457 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:42.819426    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-748b854566-dbwqz"
Nov 03 08:40:42.823122 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:42.823102    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-748b854566-dbwqz"
Nov 03 08:40:46.935703 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:40:46.935668    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:40:51.532865 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:51.532833    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 08:40:51.533339 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:51.533130    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="84d30531-a4d7-43e8-b283-4ce98cf06b9d" containerName="alertmanager" containerID="cri-o://87bedfa3971e96aa58a2edccd8bc6889eb5928eb6bb363486ca2653860354c2b" gracePeriod=120
Nov 03 08:40:51.533339 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:51.533181    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="84d30531-a4d7-43e8-b283-4ce98cf06b9d" containerName="kube-rbac-proxy-web" containerID="cri-o://b53622a4893ef5a261353f2fbe6c9950fbb88ef5cd8c7ca83ba9221aa83e90ad" gracePeriod=120
Nov 03 08:40:51.533339 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:51.533157    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="84d30531-a4d7-43e8-b283-4ce98cf06b9d" containerName="kube-rbac-proxy" containerID="cri-o://de39a77d1b8bddb52de41b7e92d3418b8bd98a6ad68483d7747fd07dfefa74df" gracePeriod=120
Nov 03 08:40:51.533339 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:51.533206    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="84d30531-a4d7-43e8-b283-4ce98cf06b9d" containerName="prom-label-proxy" containerID="cri-o://23269a172b5bec18a9d604863f59344b206b1b205ca2239130d3d645d9e1219b" gracePeriod=120
Nov 03 08:40:51.533339 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:51.533257    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="84d30531-a4d7-43e8-b283-4ce98cf06b9d" containerName="kube-rbac-proxy-metric" containerID="cri-o://62d7f13ce9b644220d38ca20d41268b0d2f8b547669da38d7bbeb7057bb69474" gracePeriod=120
Nov 03 08:40:51.533571 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:51.533468    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="84d30531-a4d7-43e8-b283-4ce98cf06b9d" containerName="config-reloader" containerID="cri-o://acd60427a52136e3bbeabda0fa0773303f0b0f058f86350638af07947c718f75" gracePeriod=120
Nov 03 08:40:52.235509 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:52.235480    4473 generic.go:356] "Generic (PLEG): container finished" podID="84d30531-a4d7-43e8-b283-4ce98cf06b9d" containerID="23269a172b5bec18a9d604863f59344b206b1b205ca2239130d3d645d9e1219b" exitCode=0
Nov 03 08:40:52.235509 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:52.235503    4473 generic.go:356] "Generic (PLEG): container finished" podID="84d30531-a4d7-43e8-b283-4ce98cf06b9d" containerID="de39a77d1b8bddb52de41b7e92d3418b8bd98a6ad68483d7747fd07dfefa74df" exitCode=0
Nov 03 08:40:52.235509 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:52.235509    4473 generic.go:356] "Generic (PLEG): container finished" podID="84d30531-a4d7-43e8-b283-4ce98cf06b9d" containerID="acd60427a52136e3bbeabda0fa0773303f0b0f058f86350638af07947c718f75" exitCode=0
Nov 03 08:40:52.235509 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:52.235514    4473 generic.go:356] "Generic (PLEG): container finished" podID="84d30531-a4d7-43e8-b283-4ce98cf06b9d" containerID="87bedfa3971e96aa58a2edccd8bc6889eb5928eb6bb363486ca2653860354c2b" exitCode=0
Nov 03 08:40:52.235775 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:52.235561    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84d30531-a4d7-43e8-b283-4ce98cf06b9d","Type":"ContainerDied","Data":"23269a172b5bec18a9d604863f59344b206b1b205ca2239130d3d645d9e1219b"}
Nov 03 08:40:52.235775 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:52.235631    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84d30531-a4d7-43e8-b283-4ce98cf06b9d","Type":"ContainerDied","Data":"de39a77d1b8bddb52de41b7e92d3418b8bd98a6ad68483d7747fd07dfefa74df"}
Nov 03 08:40:52.235775 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:52.235659    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84d30531-a4d7-43e8-b283-4ce98cf06b9d","Type":"ContainerDied","Data":"acd60427a52136e3bbeabda0fa0773303f0b0f058f86350638af07947c718f75"}
Nov 03 08:40:52.235775 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:52.235685    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84d30531-a4d7-43e8-b283-4ce98cf06b9d","Type":"ContainerDied","Data":"87bedfa3971e96aa58a2edccd8bc6889eb5928eb6bb363486ca2653860354c2b"}
Nov 03 08:40:52.786832 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:52.786809    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:52.908071 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:52.908042    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") "
Nov 03 08:40:52.908241 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:52.908077    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-cluster-tls-config\") pod \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") "
Nov 03 08:40:52.908241 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:52.908111    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-secret-alertmanager-kube-rbac-proxy-web\") pod \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") "
Nov 03 08:40:52.908241 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:52.908152    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84d30531-a4d7-43e8-b283-4ce98cf06b9d-config-out\" (UniqueName: \"kubernetes.io/empty-dir/84d30531-a4d7-43e8-b283-4ce98cf06b9d-config-out\") pod \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") "
Nov 03 08:40:52.908241 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:52.908181    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-config-volume\" (UniqueName: \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-config-volume\") pod \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") "
Nov 03 08:40:52.908241 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:52.908208    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84d30531-a4d7-43e8-b283-4ce98cf06b9d-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84d30531-a4d7-43e8-b283-4ce98cf06b9d-metrics-client-ca\") pod \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") "
Nov 03 08:40:52.908495 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:52.908302    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-web-config\" (UniqueName: \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-web-config\") pod \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") "
Nov 03 08:40:52.908495 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:52.908352    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-secret-alertmanager-main-tls\") pod \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") "
Nov 03 08:40:52.908495 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:52.908414    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84d30531-a4d7-43e8-b283-4ce98cf06b9d-kube-api-access-2bl8g\" (UniqueName: \"kubernetes.io/projected/84d30531-a4d7-43e8-b283-4ce98cf06b9d-kube-api-access-2bl8g\") pod \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") "
Nov 03 08:40:52.908495 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:52.908473    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-secret-alertmanager-kube-rbac-proxy\") pod \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") "
Nov 03 08:40:52.908685 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:52.908507    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84d30531-a4d7-43e8-b283-4ce98cf06b9d-tls-assets\" (UniqueName: \"kubernetes.io/projected/84d30531-a4d7-43e8-b283-4ce98cf06b9d-tls-assets\") pod \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") "
Nov 03 08:40:52.908685 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:52.908545    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84d30531-a4d7-43e8-b283-4ce98cf06b9d-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/84d30531-a4d7-43e8-b283-4ce98cf06b9d-alertmanager-main-db\") pod \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") "
Nov 03 08:40:52.908685 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:52.908569    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84d30531-a4d7-43e8-b283-4ce98cf06b9d-metrics-client-ca" pod "84d30531-a4d7-43e8-b283-4ce98cf06b9d" (UID: "84d30531-a4d7-43e8-b283-4ce98cf06b9d"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:40:52.908685 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:52.908574    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84d30531-a4d7-43e8-b283-4ce98cf06b9d-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84d30531-a4d7-43e8-b283-4ce98cf06b9d-alertmanager-trusted-ca-bundle\") pod \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\" (UID: \"84d30531-a4d7-43e8-b283-4ce98cf06b9d\") "
Nov 03 08:40:52.908880 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:52.908859    4473 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84d30531-a4d7-43e8-b283-4ce98cf06b9d-metrics-client-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:40:52.909132 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:52.909105    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84d30531-a4d7-43e8-b283-4ce98cf06b9d-alertmanager-trusted-ca-bundle" pod "84d30531-a4d7-43e8-b283-4ce98cf06b9d" (UID: "84d30531-a4d7-43e8-b283-4ce98cf06b9d"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:40:52.909815 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:52.909785    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84d30531-a4d7-43e8-b283-4ce98cf06b9d-alertmanager-main-db" pod "84d30531-a4d7-43e8-b283-4ce98cf06b9d" (UID: "84d30531-a4d7-43e8-b283-4ce98cf06b9d"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:40:52.912021 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:52.911999    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-secret-alertmanager-kube-rbac-proxy-metric" pod "84d30531-a4d7-43e8-b283-4ce98cf06b9d" (UID: "84d30531-a4d7-43e8-b283-4ce98cf06b9d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:40:52.912669 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:52.912638    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-secret-alertmanager-kube-rbac-proxy-web" pod "84d30531-a4d7-43e8-b283-4ce98cf06b9d" (UID: "84d30531-a4d7-43e8-b283-4ce98cf06b9d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:40:52.912669 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:52.912650    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-secret-alertmanager-kube-rbac-proxy" pod "84d30531-a4d7-43e8-b283-4ce98cf06b9d" (UID: "84d30531-a4d7-43e8-b283-4ce98cf06b9d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:40:52.912839 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:52.912699    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84d30531-a4d7-43e8-b283-4ce98cf06b9d-config-out" pod "84d30531-a4d7-43e8-b283-4ce98cf06b9d" (UID: "84d30531-a4d7-43e8-b283-4ce98cf06b9d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:40:52.913246 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:52.913225    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-config-volume" pod "84d30531-a4d7-43e8-b283-4ce98cf06b9d" (UID: "84d30531-a4d7-43e8-b283-4ce98cf06b9d"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:40:52.913328 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:52.913247    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84d30531-a4d7-43e8-b283-4ce98cf06b9d-kube-api-access-2bl8g" pod "84d30531-a4d7-43e8-b283-4ce98cf06b9d" (UID: "84d30531-a4d7-43e8-b283-4ce98cf06b9d"). InnerVolumeSpecName "kube-api-access-2bl8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:40:52.913328 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:52.913247    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-secret-alertmanager-main-tls" pod "84d30531-a4d7-43e8-b283-4ce98cf06b9d" (UID: "84d30531-a4d7-43e8-b283-4ce98cf06b9d"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:40:52.913328 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:52.913268    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84d30531-a4d7-43e8-b283-4ce98cf06b9d-tls-assets" pod "84d30531-a4d7-43e8-b283-4ce98cf06b9d" (UID: "84d30531-a4d7-43e8-b283-4ce98cf06b9d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:40:52.916514 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:52.916372    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-cluster-tls-config" pod "84d30531-a4d7-43e8-b283-4ce98cf06b9d" (UID: "84d30531-a4d7-43e8-b283-4ce98cf06b9d"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:40:52.922634 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:52.922603    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-web-config" pod "84d30531-a4d7-43e8-b283-4ce98cf06b9d" (UID: "84d30531-a4d7-43e8-b283-4ce98cf06b9d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:40:53.009224 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.009197    4473 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/84d30531-a4d7-43e8-b283-4ce98cf06b9d-alertmanager-main-db\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:40:53.009224 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.009220    4473 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/84d30531-a4d7-43e8-b283-4ce98cf06b9d-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:40:53.009380 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.009232    4473 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:40:53.009380 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.009242    4473 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-cluster-tls-config\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:40:53.009380 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.009250    4473 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:40:53.009380 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.009259    4473 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/84d30531-a4d7-43e8-b283-4ce98cf06b9d-config-out\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:40:53.009380 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.009268    4473 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:40:53.009380 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.009276    4473 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-web-config\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:40:53.009380 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.009284    4473 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-secret-alertmanager-main-tls\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:40:53.009380 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.009292    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bl8g\" (UniqueName: \"kubernetes.io/projected/84d30531-a4d7-43e8-b283-4ce98cf06b9d-kube-api-access-2bl8g\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:40:53.009380 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.009300    4473 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/84d30531-a4d7-43e8-b283-4ce98cf06b9d-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:40:53.009380 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.009308    4473 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/84d30531-a4d7-43e8-b283-4ce98cf06b9d-tls-assets\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:40:53.246305 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.246221    4473 generic.go:356] "Generic (PLEG): container finished" podID="84d30531-a4d7-43e8-b283-4ce98cf06b9d" containerID="62d7f13ce9b644220d38ca20d41268b0d2f8b547669da38d7bbeb7057bb69474" exitCode=0
Nov 03 08:40:53.246305 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.246246    4473 generic.go:356] "Generic (PLEG): container finished" podID="84d30531-a4d7-43e8-b283-4ce98cf06b9d" containerID="b53622a4893ef5a261353f2fbe6c9950fbb88ef5cd8c7ca83ba9221aa83e90ad" exitCode=0
Nov 03 08:40:53.246305 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.246278    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84d30531-a4d7-43e8-b283-4ce98cf06b9d","Type":"ContainerDied","Data":"62d7f13ce9b644220d38ca20d41268b0d2f8b547669da38d7bbeb7057bb69474"}
Nov 03 08:40:53.246508 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.246311    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.246508 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.246354    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84d30531-a4d7-43e8-b283-4ce98cf06b9d","Type":"ContainerDied","Data":"b53622a4893ef5a261353f2fbe6c9950fbb88ef5cd8c7ca83ba9221aa83e90ad"}
Nov 03 08:40:53.246508 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.246381    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"84d30531-a4d7-43e8-b283-4ce98cf06b9d","Type":"ContainerDied","Data":"21cfe31eb73d5a47a14a9072ac52b5198d213f3cd2c6a4e42648bd72734a7759"}
Nov 03 08:40:53.246508 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.246417    4473 scope.go:122] "RemoveContainer" containerID="23269a172b5bec18a9d604863f59344b206b1b205ca2239130d3d645d9e1219b"
Nov 03 08:40:53.254294 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.254274    4473 scope.go:122] "RemoveContainer" containerID="62d7f13ce9b644220d38ca20d41268b0d2f8b547669da38d7bbeb7057bb69474"
Nov 03 08:40:53.261238 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.261221    4473 scope.go:122] "RemoveContainer" containerID="de39a77d1b8bddb52de41b7e92d3418b8bd98a6ad68483d7747fd07dfefa74df"
Nov 03 08:40:53.267532 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.267515    4473 scope.go:122] "RemoveContainer" containerID="b53622a4893ef5a261353f2fbe6c9950fbb88ef5cd8c7ca83ba9221aa83e90ad"
Nov 03 08:40:53.274246 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.274232    4473 scope.go:122] "RemoveContainer" containerID="acd60427a52136e3bbeabda0fa0773303f0b0f058f86350638af07947c718f75"
Nov 03 08:40:53.280659 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.280642    4473 scope.go:122] "RemoveContainer" containerID="87bedfa3971e96aa58a2edccd8bc6889eb5928eb6bb363486ca2653860354c2b"
Nov 03 08:40:53.287395 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.287379    4473 scope.go:122] "RemoveContainer" containerID="9f492213da003b16ff5cb0a12a1e69c58433d7b14fc186ee12edc1fc3c57245b"
Nov 03 08:40:53.293883 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.293870    4473 scope.go:122] "RemoveContainer" containerID="23269a172b5bec18a9d604863f59344b206b1b205ca2239130d3d645d9e1219b"
Nov 03 08:40:53.294151 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:40:53.294133    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23269a172b5bec18a9d604863f59344b206b1b205ca2239130d3d645d9e1219b\": container with ID starting with 23269a172b5bec18a9d604863f59344b206b1b205ca2239130d3d645d9e1219b not found: ID does not exist" containerID="23269a172b5bec18a9d604863f59344b206b1b205ca2239130d3d645d9e1219b"
Nov 03 08:40:53.294196 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.294160    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23269a172b5bec18a9d604863f59344b206b1b205ca2239130d3d645d9e1219b"} err="failed to get container status \"23269a172b5bec18a9d604863f59344b206b1b205ca2239130d3d645d9e1219b\": rpc error: code = NotFound desc = could not find container \"23269a172b5bec18a9d604863f59344b206b1b205ca2239130d3d645d9e1219b\": container with ID starting with 23269a172b5bec18a9d604863f59344b206b1b205ca2239130d3d645d9e1219b not found: ID does not exist"
Nov 03 08:40:53.294196 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.294190    4473 scope.go:122] "RemoveContainer" containerID="62d7f13ce9b644220d38ca20d41268b0d2f8b547669da38d7bbeb7057bb69474"
Nov 03 08:40:53.294399 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:40:53.294380    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62d7f13ce9b644220d38ca20d41268b0d2f8b547669da38d7bbeb7057bb69474\": container with ID starting with 62d7f13ce9b644220d38ca20d41268b0d2f8b547669da38d7bbeb7057bb69474 not found: ID does not exist" containerID="62d7f13ce9b644220d38ca20d41268b0d2f8b547669da38d7bbeb7057bb69474"
Nov 03 08:40:53.294439 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.294406    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62d7f13ce9b644220d38ca20d41268b0d2f8b547669da38d7bbeb7057bb69474"} err="failed to get container status \"62d7f13ce9b644220d38ca20d41268b0d2f8b547669da38d7bbeb7057bb69474\": rpc error: code = NotFound desc = could not find container \"62d7f13ce9b644220d38ca20d41268b0d2f8b547669da38d7bbeb7057bb69474\": container with ID starting with 62d7f13ce9b644220d38ca20d41268b0d2f8b547669da38d7bbeb7057bb69474 not found: ID does not exist"
Nov 03 08:40:53.294439 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.294421    4473 scope.go:122] "RemoveContainer" containerID="de39a77d1b8bddb52de41b7e92d3418b8bd98a6ad68483d7747fd07dfefa74df"
Nov 03 08:40:53.294625 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:40:53.294607    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de39a77d1b8bddb52de41b7e92d3418b8bd98a6ad68483d7747fd07dfefa74df\": container with ID starting with de39a77d1b8bddb52de41b7e92d3418b8bd98a6ad68483d7747fd07dfefa74df not found: ID does not exist" containerID="de39a77d1b8bddb52de41b7e92d3418b8bd98a6ad68483d7747fd07dfefa74df"
Nov 03 08:40:53.294659 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.294632    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de39a77d1b8bddb52de41b7e92d3418b8bd98a6ad68483d7747fd07dfefa74df"} err="failed to get container status \"de39a77d1b8bddb52de41b7e92d3418b8bd98a6ad68483d7747fd07dfefa74df\": rpc error: code = NotFound desc = could not find container \"de39a77d1b8bddb52de41b7e92d3418b8bd98a6ad68483d7747fd07dfefa74df\": container with ID starting with de39a77d1b8bddb52de41b7e92d3418b8bd98a6ad68483d7747fd07dfefa74df not found: ID does not exist"
Nov 03 08:40:53.294659 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.294646    4473 scope.go:122] "RemoveContainer" containerID="b53622a4893ef5a261353f2fbe6c9950fbb88ef5cd8c7ca83ba9221aa83e90ad"
Nov 03 08:40:53.294859 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:40:53.294843    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b53622a4893ef5a261353f2fbe6c9950fbb88ef5cd8c7ca83ba9221aa83e90ad\": container with ID starting with b53622a4893ef5a261353f2fbe6c9950fbb88ef5cd8c7ca83ba9221aa83e90ad not found: ID does not exist" containerID="b53622a4893ef5a261353f2fbe6c9950fbb88ef5cd8c7ca83ba9221aa83e90ad"
Nov 03 08:40:53.294916 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.294863    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b53622a4893ef5a261353f2fbe6c9950fbb88ef5cd8c7ca83ba9221aa83e90ad"} err="failed to get container status \"b53622a4893ef5a261353f2fbe6c9950fbb88ef5cd8c7ca83ba9221aa83e90ad\": rpc error: code = NotFound desc = could not find container \"b53622a4893ef5a261353f2fbe6c9950fbb88ef5cd8c7ca83ba9221aa83e90ad\": container with ID starting with b53622a4893ef5a261353f2fbe6c9950fbb88ef5cd8c7ca83ba9221aa83e90ad not found: ID does not exist"
Nov 03 08:40:53.294916 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.294876    4473 scope.go:122] "RemoveContainer" containerID="acd60427a52136e3bbeabda0fa0773303f0b0f058f86350638af07947c718f75"
Nov 03 08:40:53.295099 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:40:53.295082    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acd60427a52136e3bbeabda0fa0773303f0b0f058f86350638af07947c718f75\": container with ID starting with acd60427a52136e3bbeabda0fa0773303f0b0f058f86350638af07947c718f75 not found: ID does not exist" containerID="acd60427a52136e3bbeabda0fa0773303f0b0f058f86350638af07947c718f75"
Nov 03 08:40:53.295131 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.295105    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acd60427a52136e3bbeabda0fa0773303f0b0f058f86350638af07947c718f75"} err="failed to get container status \"acd60427a52136e3bbeabda0fa0773303f0b0f058f86350638af07947c718f75\": rpc error: code = NotFound desc = could not find container \"acd60427a52136e3bbeabda0fa0773303f0b0f058f86350638af07947c718f75\": container with ID starting with acd60427a52136e3bbeabda0fa0773303f0b0f058f86350638af07947c718f75 not found: ID does not exist"
Nov 03 08:40:53.295131 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.295121    4473 scope.go:122] "RemoveContainer" containerID="87bedfa3971e96aa58a2edccd8bc6889eb5928eb6bb363486ca2653860354c2b"
Nov 03 08:40:53.295331 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:40:53.295317    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87bedfa3971e96aa58a2edccd8bc6889eb5928eb6bb363486ca2653860354c2b\": container with ID starting with 87bedfa3971e96aa58a2edccd8bc6889eb5928eb6bb363486ca2653860354c2b not found: ID does not exist" containerID="87bedfa3971e96aa58a2edccd8bc6889eb5928eb6bb363486ca2653860354c2b"
Nov 03 08:40:53.295371 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.295333    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87bedfa3971e96aa58a2edccd8bc6889eb5928eb6bb363486ca2653860354c2b"} err="failed to get container status \"87bedfa3971e96aa58a2edccd8bc6889eb5928eb6bb363486ca2653860354c2b\": rpc error: code = NotFound desc = could not find container \"87bedfa3971e96aa58a2edccd8bc6889eb5928eb6bb363486ca2653860354c2b\": container with ID starting with 87bedfa3971e96aa58a2edccd8bc6889eb5928eb6bb363486ca2653860354c2b not found: ID does not exist"
Nov 03 08:40:53.295371 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.295344    4473 scope.go:122] "RemoveContainer" containerID="9f492213da003b16ff5cb0a12a1e69c58433d7b14fc186ee12edc1fc3c57245b"
Nov 03 08:40:53.295585 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:40:53.295568    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f492213da003b16ff5cb0a12a1e69c58433d7b14fc186ee12edc1fc3c57245b\": container with ID starting with 9f492213da003b16ff5cb0a12a1e69c58433d7b14fc186ee12edc1fc3c57245b not found: ID does not exist" containerID="9f492213da003b16ff5cb0a12a1e69c58433d7b14fc186ee12edc1fc3c57245b"
Nov 03 08:40:53.295628 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.295588    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f492213da003b16ff5cb0a12a1e69c58433d7b14fc186ee12edc1fc3c57245b"} err="failed to get container status \"9f492213da003b16ff5cb0a12a1e69c58433d7b14fc186ee12edc1fc3c57245b\": rpc error: code = NotFound desc = could not find container \"9f492213da003b16ff5cb0a12a1e69c58433d7b14fc186ee12edc1fc3c57245b\": container with ID starting with 9f492213da003b16ff5cb0a12a1e69c58433d7b14fc186ee12edc1fc3c57245b not found: ID does not exist"
Nov 03 08:40:53.295628 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.295601    4473 scope.go:122] "RemoveContainer" containerID="23269a172b5bec18a9d604863f59344b206b1b205ca2239130d3d645d9e1219b"
Nov 03 08:40:53.295810 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.295793    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23269a172b5bec18a9d604863f59344b206b1b205ca2239130d3d645d9e1219b"} err="failed to get container status \"23269a172b5bec18a9d604863f59344b206b1b205ca2239130d3d645d9e1219b\": rpc error: code = NotFound desc = could not find container \"23269a172b5bec18a9d604863f59344b206b1b205ca2239130d3d645d9e1219b\": container with ID starting with 23269a172b5bec18a9d604863f59344b206b1b205ca2239130d3d645d9e1219b not found: ID does not exist"
Nov 03 08:40:53.295852 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.295811    4473 scope.go:122] "RemoveContainer" containerID="62d7f13ce9b644220d38ca20d41268b0d2f8b547669da38d7bbeb7057bb69474"
Nov 03 08:40:53.296021 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.296005    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62d7f13ce9b644220d38ca20d41268b0d2f8b547669da38d7bbeb7057bb69474"} err="failed to get container status \"62d7f13ce9b644220d38ca20d41268b0d2f8b547669da38d7bbeb7057bb69474\": rpc error: code = NotFound desc = could not find container \"62d7f13ce9b644220d38ca20d41268b0d2f8b547669da38d7bbeb7057bb69474\": container with ID starting with 62d7f13ce9b644220d38ca20d41268b0d2f8b547669da38d7bbeb7057bb69474 not found: ID does not exist"
Nov 03 08:40:53.296021 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.296021    4473 scope.go:122] "RemoveContainer" containerID="de39a77d1b8bddb52de41b7e92d3418b8bd98a6ad68483d7747fd07dfefa74df"
Nov 03 08:40:53.296249 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.296231    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de39a77d1b8bddb52de41b7e92d3418b8bd98a6ad68483d7747fd07dfefa74df"} err="failed to get container status \"de39a77d1b8bddb52de41b7e92d3418b8bd98a6ad68483d7747fd07dfefa74df\": rpc error: code = NotFound desc = could not find container \"de39a77d1b8bddb52de41b7e92d3418b8bd98a6ad68483d7747fd07dfefa74df\": container with ID starting with de39a77d1b8bddb52de41b7e92d3418b8bd98a6ad68483d7747fd07dfefa74df not found: ID does not exist"
Nov 03 08:40:53.296298 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.296249    4473 scope.go:122] "RemoveContainer" containerID="b53622a4893ef5a261353f2fbe6c9950fbb88ef5cd8c7ca83ba9221aa83e90ad"
Nov 03 08:40:53.296481 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.296464    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b53622a4893ef5a261353f2fbe6c9950fbb88ef5cd8c7ca83ba9221aa83e90ad"} err="failed to get container status \"b53622a4893ef5a261353f2fbe6c9950fbb88ef5cd8c7ca83ba9221aa83e90ad\": rpc error: code = NotFound desc = could not find container \"b53622a4893ef5a261353f2fbe6c9950fbb88ef5cd8c7ca83ba9221aa83e90ad\": container with ID starting with b53622a4893ef5a261353f2fbe6c9950fbb88ef5cd8c7ca83ba9221aa83e90ad not found: ID does not exist"
Nov 03 08:40:53.296522 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.296483    4473 scope.go:122] "RemoveContainer" containerID="acd60427a52136e3bbeabda0fa0773303f0b0f058f86350638af07947c718f75"
Nov 03 08:40:53.296701 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.296684    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acd60427a52136e3bbeabda0fa0773303f0b0f058f86350638af07947c718f75"} err="failed to get container status \"acd60427a52136e3bbeabda0fa0773303f0b0f058f86350638af07947c718f75\": rpc error: code = NotFound desc = could not find container \"acd60427a52136e3bbeabda0fa0773303f0b0f058f86350638af07947c718f75\": container with ID starting with acd60427a52136e3bbeabda0fa0773303f0b0f058f86350638af07947c718f75 not found: ID does not exist"
Nov 03 08:40:53.296701 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.296701    4473 scope.go:122] "RemoveContainer" containerID="87bedfa3971e96aa58a2edccd8bc6889eb5928eb6bb363486ca2653860354c2b"
Nov 03 08:40:53.296881 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.296858    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87bedfa3971e96aa58a2edccd8bc6889eb5928eb6bb363486ca2653860354c2b"} err="failed to get container status \"87bedfa3971e96aa58a2edccd8bc6889eb5928eb6bb363486ca2653860354c2b\": rpc error: code = NotFound desc = could not find container \"87bedfa3971e96aa58a2edccd8bc6889eb5928eb6bb363486ca2653860354c2b\": container with ID starting with 87bedfa3971e96aa58a2edccd8bc6889eb5928eb6bb363486ca2653860354c2b not found: ID does not exist"
Nov 03 08:40:53.296931 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.296882    4473 scope.go:122] "RemoveContainer" containerID="9f492213da003b16ff5cb0a12a1e69c58433d7b14fc186ee12edc1fc3c57245b"
Nov 03 08:40:53.297138 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.297122    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f492213da003b16ff5cb0a12a1e69c58433d7b14fc186ee12edc1fc3c57245b"} err="failed to get container status \"9f492213da003b16ff5cb0a12a1e69c58433d7b14fc186ee12edc1fc3c57245b\": rpc error: code = NotFound desc = could not find container \"9f492213da003b16ff5cb0a12a1e69c58433d7b14fc186ee12edc1fc3c57245b\": container with ID starting with 9f492213da003b16ff5cb0a12a1e69c58433d7b14fc186ee12edc1fc3c57245b not found: ID does not exist"
Nov 03 08:40:53.417240 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.417212    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 08:40:53.423596 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.423577    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 08:40:53.467628 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.467601    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 08:40:53.467890 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.467876    4473 cpu_manager.go:414] "containerMap: removing container" podUID="84d30531-a4d7-43e8-b283-4ce98cf06b9d" containerName="kube-rbac-proxy"
Nov 03 08:40:53.467968 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.467892    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84d30531-a4d7-43e8-b283-4ce98cf06b9d" containerName="kube-rbac-proxy"
Nov 03 08:40:53.467968 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.467906    4473 cpu_manager.go:414] "containerMap: removing container" podUID="84d30531-a4d7-43e8-b283-4ce98cf06b9d" containerName="alertmanager"
Nov 03 08:40:53.467968 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.467911    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84d30531-a4d7-43e8-b283-4ce98cf06b9d" containerName="alertmanager"
Nov 03 08:40:53.467968 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.467918    4473 cpu_manager.go:414] "containerMap: removing container" podUID="84d30531-a4d7-43e8-b283-4ce98cf06b9d" containerName="prom-label-proxy"
Nov 03 08:40:53.467968 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.467922    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84d30531-a4d7-43e8-b283-4ce98cf06b9d" containerName="prom-label-proxy"
Nov 03 08:40:53.467968 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.467930    4473 cpu_manager.go:414] "containerMap: removing container" podUID="84d30531-a4d7-43e8-b283-4ce98cf06b9d" containerName="kube-rbac-proxy-web"
Nov 03 08:40:53.467968 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.467934    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84d30531-a4d7-43e8-b283-4ce98cf06b9d" containerName="kube-rbac-proxy-web"
Nov 03 08:40:53.467968 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.467940    4473 cpu_manager.go:414] "containerMap: removing container" podUID="84d30531-a4d7-43e8-b283-4ce98cf06b9d" containerName="kube-rbac-proxy-metric"
Nov 03 08:40:53.467968 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.467963    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84d30531-a4d7-43e8-b283-4ce98cf06b9d" containerName="kube-rbac-proxy-metric"
Nov 03 08:40:53.468266 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.467979    4473 cpu_manager.go:414] "containerMap: removing container" podUID="84d30531-a4d7-43e8-b283-4ce98cf06b9d" containerName="init-config-reloader"
Nov 03 08:40:53.468266 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.467984    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84d30531-a4d7-43e8-b283-4ce98cf06b9d" containerName="init-config-reloader"
Nov 03 08:40:53.468266 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.467992    4473 cpu_manager.go:414] "containerMap: removing container" podUID="84d30531-a4d7-43e8-b283-4ce98cf06b9d" containerName="config-reloader"
Nov 03 08:40:53.468266 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.467996    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84d30531-a4d7-43e8-b283-4ce98cf06b9d" containerName="config-reloader"
Nov 03 08:40:53.468266 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.468042    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="84d30531-a4d7-43e8-b283-4ce98cf06b9d" containerName="config-reloader"
Nov 03 08:40:53.468266 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.468055    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="84d30531-a4d7-43e8-b283-4ce98cf06b9d" containerName="kube-rbac-proxy"
Nov 03 08:40:53.468266 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.468063    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="84d30531-a4d7-43e8-b283-4ce98cf06b9d" containerName="alertmanager"
Nov 03 08:40:53.468266 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.468067    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="84d30531-a4d7-43e8-b283-4ce98cf06b9d" containerName="prom-label-proxy"
Nov 03 08:40:53.468266 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.468072    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="84d30531-a4d7-43e8-b283-4ce98cf06b9d" containerName="kube-rbac-proxy-web"
Nov 03 08:40:53.468266 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.468077    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="84d30531-a4d7-43e8-b283-4ce98cf06b9d" containerName="kube-rbac-proxy-metric"
Nov 03 08:40:53.472826 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.472811    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.528432 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.528366    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 08:40:53.586908 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.586846    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 08:40:53.586908 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.586845    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 08:40:53.586908 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.586902    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 08:40:53.587171 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.586908    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 08:40:53.587171 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.586963    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 08:40:53.587171 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.586910    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 08:40:53.587360 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.587274    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 08:40:53.587862 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.587434    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 08:40:53.587862 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.587537    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-g2c94\""
Nov 03 08:40:53.589225 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.589207    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 08:40:53.613418 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.613396    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e5f3c403-9baf-4ac8-8524-88513a5b7d1c-config-out\") pod \"alertmanager-main-0\" (UID: \"e5f3c403-9baf-4ac8-8524-88513a5b7d1c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.613514 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.613430    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e5f3c403-9baf-4ac8-8524-88513a5b7d1c-config-volume\") pod \"alertmanager-main-0\" (UID: \"e5f3c403-9baf-4ac8-8524-88513a5b7d1c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.613514 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.613456    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e5f3c403-9baf-4ac8-8524-88513a5b7d1c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e5f3c403-9baf-4ac8-8524-88513a5b7d1c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.613514 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.613481    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e5f3c403-9baf-4ac8-8524-88513a5b7d1c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e5f3c403-9baf-4ac8-8524-88513a5b7d1c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.613640 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.613555    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5f3c403-9baf-4ac8-8524-88513a5b7d1c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e5f3c403-9baf-4ac8-8524-88513a5b7d1c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.613640 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.613588    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e5f3c403-9baf-4ac8-8524-88513a5b7d1c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e5f3c403-9baf-4ac8-8524-88513a5b7d1c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.613640 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.613609    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e5f3c403-9baf-4ac8-8524-88513a5b7d1c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e5f3c403-9baf-4ac8-8524-88513a5b7d1c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.613802 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.613710    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e5f3c403-9baf-4ac8-8524-88513a5b7d1c-web-config\") pod \"alertmanager-main-0\" (UID: \"e5f3c403-9baf-4ac8-8524-88513a5b7d1c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.613802 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.613742    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e5f3c403-9baf-4ac8-8524-88513a5b7d1c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e5f3c403-9baf-4ac8-8524-88513a5b7d1c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.613802 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.613781    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lwz7\" (UniqueName: \"kubernetes.io/projected/e5f3c403-9baf-4ac8-8524-88513a5b7d1c-kube-api-access-7lwz7\") pod \"alertmanager-main-0\" (UID: \"e5f3c403-9baf-4ac8-8524-88513a5b7d1c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.613902 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.613806    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e5f3c403-9baf-4ac8-8524-88513a5b7d1c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e5f3c403-9baf-4ac8-8524-88513a5b7d1c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.613902 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.613824    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e5f3c403-9baf-4ac8-8524-88513a5b7d1c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e5f3c403-9baf-4ac8-8524-88513a5b7d1c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.613902 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.613881    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5f3c403-9baf-4ac8-8524-88513a5b7d1c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e5f3c403-9baf-4ac8-8524-88513a5b7d1c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.714740 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.714710    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e5f3c403-9baf-4ac8-8524-88513a5b7d1c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e5f3c403-9baf-4ac8-8524-88513a5b7d1c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.714881 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.714745    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5f3c403-9baf-4ac8-8524-88513a5b7d1c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e5f3c403-9baf-4ac8-8524-88513a5b7d1c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.714881 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.714772    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e5f3c403-9baf-4ac8-8524-88513a5b7d1c-config-out\") pod \"alertmanager-main-0\" (UID: \"e5f3c403-9baf-4ac8-8524-88513a5b7d1c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.714881 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.714793    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e5f3c403-9baf-4ac8-8524-88513a5b7d1c-config-volume\") pod \"alertmanager-main-0\" (UID: \"e5f3c403-9baf-4ac8-8524-88513a5b7d1c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.714881 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.714852    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e5f3c403-9baf-4ac8-8524-88513a5b7d1c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e5f3c403-9baf-4ac8-8524-88513a5b7d1c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.714881 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.714880    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e5f3c403-9baf-4ac8-8524-88513a5b7d1c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e5f3c403-9baf-4ac8-8524-88513a5b7d1c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.715144 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.714901    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5f3c403-9baf-4ac8-8524-88513a5b7d1c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e5f3c403-9baf-4ac8-8524-88513a5b7d1c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.715144 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.714954    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e5f3c403-9baf-4ac8-8524-88513a5b7d1c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e5f3c403-9baf-4ac8-8524-88513a5b7d1c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.715144 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.714989    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e5f3c403-9baf-4ac8-8524-88513a5b7d1c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e5f3c403-9baf-4ac8-8524-88513a5b7d1c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.715144 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.715020    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e5f3c403-9baf-4ac8-8524-88513a5b7d1c-web-config\") pod \"alertmanager-main-0\" (UID: \"e5f3c403-9baf-4ac8-8524-88513a5b7d1c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.715144 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.715048    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e5f3c403-9baf-4ac8-8524-88513a5b7d1c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e5f3c403-9baf-4ac8-8524-88513a5b7d1c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.715144 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.715080    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lwz7\" (UniqueName: \"kubernetes.io/projected/e5f3c403-9baf-4ac8-8524-88513a5b7d1c-kube-api-access-7lwz7\") pod \"alertmanager-main-0\" (UID: \"e5f3c403-9baf-4ac8-8524-88513a5b7d1c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.715144 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.715114    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e5f3c403-9baf-4ac8-8524-88513a5b7d1c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e5f3c403-9baf-4ac8-8524-88513a5b7d1c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.715786 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.715754    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5f3c403-9baf-4ac8-8524-88513a5b7d1c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e5f3c403-9baf-4ac8-8524-88513a5b7d1c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.716249 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.715906    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5f3c403-9baf-4ac8-8524-88513a5b7d1c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e5f3c403-9baf-4ac8-8524-88513a5b7d1c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.716249 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.716207    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e5f3c403-9baf-4ac8-8524-88513a5b7d1c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e5f3c403-9baf-4ac8-8524-88513a5b7d1c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.718254 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.718228    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e5f3c403-9baf-4ac8-8524-88513a5b7d1c-config-volume\") pod \"alertmanager-main-0\" (UID: \"e5f3c403-9baf-4ac8-8524-88513a5b7d1c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.718364 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.718237    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e5f3c403-9baf-4ac8-8524-88513a5b7d1c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e5f3c403-9baf-4ac8-8524-88513a5b7d1c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.718793 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.718748    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e5f3c403-9baf-4ac8-8524-88513a5b7d1c-config-out\") pod \"alertmanager-main-0\" (UID: \"e5f3c403-9baf-4ac8-8524-88513a5b7d1c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.718921 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.718900    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e5f3c403-9baf-4ac8-8524-88513a5b7d1c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e5f3c403-9baf-4ac8-8524-88513a5b7d1c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.718999 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.718906    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e5f3c403-9baf-4ac8-8524-88513a5b7d1c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e5f3c403-9baf-4ac8-8524-88513a5b7d1c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.719060 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.719044    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e5f3c403-9baf-4ac8-8524-88513a5b7d1c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e5f3c403-9baf-4ac8-8524-88513a5b7d1c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.719111 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.719071    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e5f3c403-9baf-4ac8-8524-88513a5b7d1c-web-config\") pod \"alertmanager-main-0\" (UID: \"e5f3c403-9baf-4ac8-8524-88513a5b7d1c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.719640 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.719615    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e5f3c403-9baf-4ac8-8524-88513a5b7d1c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e5f3c403-9baf-4ac8-8524-88513a5b7d1c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.720157 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.720141    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e5f3c403-9baf-4ac8-8524-88513a5b7d1c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e5f3c403-9baf-4ac8-8524-88513a5b7d1c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.776648 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.776625    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lwz7\" (UniqueName: \"kubernetes.io/projected/e5f3c403-9baf-4ac8-8524-88513a5b7d1c-kube-api-access-7lwz7\") pod \"alertmanager-main-0\" (UID: \"e5f3c403-9baf-4ac8-8524-88513a5b7d1c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.795984 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.795917    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 08:40:53.868251 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:53.868221    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84d30531-a4d7-43e8-b283-4ce98cf06b9d" path="/var/lib/kubelet/pods/84d30531-a4d7-43e8-b283-4ce98cf06b9d/volumes"
Nov 03 08:40:54.224717 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:54.224688    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 08:40:54.227568 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:40:54.227540    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5f3c403_9baf_4ac8_8524_88513a5b7d1c.slice/crio-0765cf2fc7be3eba4fffe7fe61e2d95e07cc2728e22770187bbecbbe41d3bcdd WatchSource:0}: Error finding container 0765cf2fc7be3eba4fffe7fe61e2d95e07cc2728e22770187bbecbbe41d3bcdd: Status 404 returned error can't find the container with id 0765cf2fc7be3eba4fffe7fe61e2d95e07cc2728e22770187bbecbbe41d3bcdd
Nov 03 08:40:54.251354 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:54.251322    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5f3c403-9baf-4ac8-8524-88513a5b7d1c","Type":"ContainerStarted","Data":"0765cf2fc7be3eba4fffe7fe61e2d95e07cc2728e22770187bbecbbe41d3bcdd"}
Nov 03 08:40:55.142017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:55.141984    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6d8c96b47-jh99f"]
Nov 03 08:40:55.145061 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:55.145040    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6d8c96b47-jh99f"
Nov 03 08:40:55.200406 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:55.200376    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 03 08:40:55.200406 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:55.200389    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 03 08:40:55.200638 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:55.200441    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 03 08:40:55.200638 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:55.200562    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 03 08:40:55.200638 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:55.200630    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-742ck\""
Nov 03 08:40:55.224983 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:55.224935    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfbbe91f-114b-4e33-a356-511602c939e7-serving-certs-ca-bundle\") pod \"telemeter-client-6d8c96b47-jh99f\" (UID: \"bfbbe91f-114b-4e33-a356-511602c939e7\") " pod="openshift-monitoring/telemeter-client-6d8c96b47-jh99f"
Nov 03 08:40:55.225070 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:55.225009    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bfbbe91f-114b-4e33-a356-511602c939e7-federate-client-tls\") pod \"telemeter-client-6d8c96b47-jh99f\" (UID: \"bfbbe91f-114b-4e33-a356-511602c939e7\") " pod="openshift-monitoring/telemeter-client-6d8c96b47-jh99f"
Nov 03 08:40:55.225070 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:55.225052    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bfbbe91f-114b-4e33-a356-511602c939e7-secret-telemeter-client\") pod \"telemeter-client-6d8c96b47-jh99f\" (UID: \"bfbbe91f-114b-4e33-a356-511602c939e7\") " pod="openshift-monitoring/telemeter-client-6d8c96b47-jh99f"
Nov 03 08:40:55.225180 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:55.225074    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bfbbe91f-114b-4e33-a356-511602c939e7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6d8c96b47-jh99f\" (UID: \"bfbbe91f-114b-4e33-a356-511602c939e7\") " pod="openshift-monitoring/telemeter-client-6d8c96b47-jh99f"
Nov 03 08:40:55.225180 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:55.225094    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pbnh\" (UniqueName: \"kubernetes.io/projected/bfbbe91f-114b-4e33-a356-511602c939e7-kube-api-access-4pbnh\") pod \"telemeter-client-6d8c96b47-jh99f\" (UID: \"bfbbe91f-114b-4e33-a356-511602c939e7\") " pod="openshift-monitoring/telemeter-client-6d8c96b47-jh99f"
Nov 03 08:40:55.225263 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:55.225182    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfbbe91f-114b-4e33-a356-511602c939e7-metrics-client-ca\") pod \"telemeter-client-6d8c96b47-jh99f\" (UID: \"bfbbe91f-114b-4e33-a356-511602c939e7\") " pod="openshift-monitoring/telemeter-client-6d8c96b47-jh99f"
Nov 03 08:40:55.225293 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:55.225285    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bfbbe91f-114b-4e33-a356-511602c939e7-telemeter-client-tls\") pod \"telemeter-client-6d8c96b47-jh99f\" (UID: \"bfbbe91f-114b-4e33-a356-511602c939e7\") " pod="openshift-monitoring/telemeter-client-6d8c96b47-jh99f"
Nov 03 08:40:55.225331 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:55.225305    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfbbe91f-114b-4e33-a356-511602c939e7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6d8c96b47-jh99f\" (UID: \"bfbbe91f-114b-4e33-a356-511602c939e7\") " pod="openshift-monitoring/telemeter-client-6d8c96b47-jh99f"
Nov 03 08:40:55.250013 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:55.249981    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 03 08:40:55.257876 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:55.257847    4473 generic.go:356] "Generic (PLEG): container finished" podID="e5f3c403-9baf-4ac8-8524-88513a5b7d1c" containerID="b439f5607ceae1dddbf78742281d15ed10e6297e03b33bdf16867fa3a47caf59" exitCode=0
Nov 03 08:40:55.258000 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:55.257911    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5f3c403-9baf-4ac8-8524-88513a5b7d1c","Type":"ContainerDied","Data":"b439f5607ceae1dddbf78742281d15ed10e6297e03b33bdf16867fa3a47caf59"}
Nov 03 08:40:55.300790 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:55.300760    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 03 08:40:55.306585 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:55.306244    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6d8c96b47-jh99f"]
Nov 03 08:40:55.326254 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:55.326231    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bfbbe91f-114b-4e33-a356-511602c939e7-federate-client-tls\") pod \"telemeter-client-6d8c96b47-jh99f\" (UID: \"bfbbe91f-114b-4e33-a356-511602c939e7\") " pod="openshift-monitoring/telemeter-client-6d8c96b47-jh99f"
Nov 03 08:40:55.326422 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:55.326401    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bfbbe91f-114b-4e33-a356-511602c939e7-secret-telemeter-client\") pod \"telemeter-client-6d8c96b47-jh99f\" (UID: \"bfbbe91f-114b-4e33-a356-511602c939e7\") " pod="openshift-monitoring/telemeter-client-6d8c96b47-jh99f"
Nov 03 08:40:55.326504 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:55.326445    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bfbbe91f-114b-4e33-a356-511602c939e7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6d8c96b47-jh99f\" (UID: \"bfbbe91f-114b-4e33-a356-511602c939e7\") " pod="openshift-monitoring/telemeter-client-6d8c96b47-jh99f"
Nov 03 08:40:55.326504 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:55.326474    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pbnh\" (UniqueName: \"kubernetes.io/projected/bfbbe91f-114b-4e33-a356-511602c939e7-kube-api-access-4pbnh\") pod \"telemeter-client-6d8c96b47-jh99f\" (UID: \"bfbbe91f-114b-4e33-a356-511602c939e7\") " pod="openshift-monitoring/telemeter-client-6d8c96b47-jh99f"
Nov 03 08:40:55.326621 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:55.326504    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfbbe91f-114b-4e33-a356-511602c939e7-metrics-client-ca\") pod \"telemeter-client-6d8c96b47-jh99f\" (UID: \"bfbbe91f-114b-4e33-a356-511602c939e7\") " pod="openshift-monitoring/telemeter-client-6d8c96b47-jh99f"
Nov 03 08:40:55.326696 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:55.326657    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bfbbe91f-114b-4e33-a356-511602c939e7-telemeter-client-tls\") pod \"telemeter-client-6d8c96b47-jh99f\" (UID: \"bfbbe91f-114b-4e33-a356-511602c939e7\") " pod="openshift-monitoring/telemeter-client-6d8c96b47-jh99f"
Nov 03 08:40:55.326696 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:55.326690    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfbbe91f-114b-4e33-a356-511602c939e7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6d8c96b47-jh99f\" (UID: \"bfbbe91f-114b-4e33-a356-511602c939e7\") " pod="openshift-monitoring/telemeter-client-6d8c96b47-jh99f"
Nov 03 08:40:55.326821 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:55.326726    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfbbe91f-114b-4e33-a356-511602c939e7-serving-certs-ca-bundle\") pod \"telemeter-client-6d8c96b47-jh99f\" (UID: \"bfbbe91f-114b-4e33-a356-511602c939e7\") " pod="openshift-monitoring/telemeter-client-6d8c96b47-jh99f"
Nov 03 08:40:55.327594 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:55.327543    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bfbbe91f-114b-4e33-a356-511602c939e7-metrics-client-ca\") pod \"telemeter-client-6d8c96b47-jh99f\" (UID: \"bfbbe91f-114b-4e33-a356-511602c939e7\") " pod="openshift-monitoring/telemeter-client-6d8c96b47-jh99f"
Nov 03 08:40:55.327723 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:55.327699    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfbbe91f-114b-4e33-a356-511602c939e7-serving-certs-ca-bundle\") pod \"telemeter-client-6d8c96b47-jh99f\" (UID: \"bfbbe91f-114b-4e33-a356-511602c939e7\") " pod="openshift-monitoring/telemeter-client-6d8c96b47-jh99f"
Nov 03 08:40:55.328239 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:55.328219    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfbbe91f-114b-4e33-a356-511602c939e7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6d8c96b47-jh99f\" (UID: \"bfbbe91f-114b-4e33-a356-511602c939e7\") " pod="openshift-monitoring/telemeter-client-6d8c96b47-jh99f"
Nov 03 08:40:55.329672 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:55.329645    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bfbbe91f-114b-4e33-a356-511602c939e7-federate-client-tls\") pod \"telemeter-client-6d8c96b47-jh99f\" (UID: \"bfbbe91f-114b-4e33-a356-511602c939e7\") " pod="openshift-monitoring/telemeter-client-6d8c96b47-jh99f"
Nov 03 08:40:55.329982 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:55.329935    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bfbbe91f-114b-4e33-a356-511602c939e7-telemeter-client-tls\") pod \"telemeter-client-6d8c96b47-jh99f\" (UID: \"bfbbe91f-114b-4e33-a356-511602c939e7\") " pod="openshift-monitoring/telemeter-client-6d8c96b47-jh99f"
Nov 03 08:40:55.330276 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:55.330258    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bfbbe91f-114b-4e33-a356-511602c939e7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6d8c96b47-jh99f\" (UID: \"bfbbe91f-114b-4e33-a356-511602c939e7\") " pod="openshift-monitoring/telemeter-client-6d8c96b47-jh99f"
Nov 03 08:40:55.330348 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:55.330299    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bfbbe91f-114b-4e33-a356-511602c939e7-secret-telemeter-client\") pod \"telemeter-client-6d8c96b47-jh99f\" (UID: \"bfbbe91f-114b-4e33-a356-511602c939e7\") " pod="openshift-monitoring/telemeter-client-6d8c96b47-jh99f"
Nov 03 08:40:55.385751 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:55.385728    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pbnh\" (UniqueName: \"kubernetes.io/projected/bfbbe91f-114b-4e33-a356-511602c939e7-kube-api-access-4pbnh\") pod \"telemeter-client-6d8c96b47-jh99f\" (UID: \"bfbbe91f-114b-4e33-a356-511602c939e7\") " pod="openshift-monitoring/telemeter-client-6d8c96b47-jh99f"
Nov 03 08:40:55.459929 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:55.459903    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6d8c96b47-jh99f"
Nov 03 08:40:55.894582 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:55.894554    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6d8c96b47-jh99f"]
Nov 03 08:40:55.898409 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:40:55.898382    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfbbe91f_114b_4e33_a356_511602c939e7.slice/crio-5fee199c304a9eb95dc11001ff480655efe52cf8faf4fc0e21929190d28ff898 WatchSource:0}: Error finding container 5fee199c304a9eb95dc11001ff480655efe52cf8faf4fc0e21929190d28ff898: Status 404 returned error can't find the container with id 5fee199c304a9eb95dc11001ff480655efe52cf8faf4fc0e21929190d28ff898
Nov 03 08:40:56.265535 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:56.265495    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5f3c403-9baf-4ac8-8524-88513a5b7d1c","Type":"ContainerStarted","Data":"71a10b07fb0674460b43839cdf47024d0a95afcc628408c9442162c8ed0c3806"}
Nov 03 08:40:56.266015 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:56.265579    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5f3c403-9baf-4ac8-8524-88513a5b7d1c","Type":"ContainerStarted","Data":"14b536af9a98986d4b02f8c1e5e64f9462e44a6fda2ba9c747dec6048b966096"}
Nov 03 08:40:56.266015 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:56.265620    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5f3c403-9baf-4ac8-8524-88513a5b7d1c","Type":"ContainerStarted","Data":"bf3dae8fc20fbc90c8bb7533e0b6a0e9cbf3c0b4c531d009c7b46dd428648d96"}
Nov 03 08:40:56.266015 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:56.265658    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5f3c403-9baf-4ac8-8524-88513a5b7d1c","Type":"ContainerStarted","Data":"08745e8c3ab4d0257e2b8a918e053639aadf4de33f9e0581d9a78084133d53e0"}
Nov 03 08:40:56.266015 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:56.265691    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5f3c403-9baf-4ac8-8524-88513a5b7d1c","Type":"ContainerStarted","Data":"8ca6b13a8a8b051898a26fa79fef5fa9fce661b9b15da66969eca3d001a24ef6"}
Nov 03 08:40:56.266015 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:56.265726    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e5f3c403-9baf-4ac8-8524-88513a5b7d1c","Type":"ContainerStarted","Data":"11c2f19ab7e938e94bb45b6cd6f27ccbdd26ba1f9098389cc33810955b218054"}
Nov 03 08:40:56.266766 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:56.266740    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6d8c96b47-jh99f" event={"ID":"bfbbe91f-114b-4e33-a356-511602c939e7","Type":"ContainerStarted","Data":"5fee199c304a9eb95dc11001ff480655efe52cf8faf4fc0e21929190d28ff898"}
Nov 03 08:40:56.391653 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:56.391595    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.379817672 podStartE2EDuration="3.391573897s" totalImagesPullingTime="0s" totalInitContainerRuntime="11.756225ms" isStatefulPod=false podCreationTimestamp="2026-11-03 08:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:40:56.390152127 +0000 UTC m=+80.659215581" watchObservedRunningTime="2026-11-03 08:40:56.391573897 +0000 UTC m=+80.660637333"
Nov 03 08:40:58.276267 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:58.276186    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6d8c96b47-jh99f" event={"ID":"bfbbe91f-114b-4e33-a356-511602c939e7","Type":"ContainerStarted","Data":"034d138eeb3936547edfc08389ed862d5f451c310065d1a33c7272ee99cf9e46"}
Nov 03 08:40:58.276267 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:58.276245    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6d8c96b47-jh99f" event={"ID":"bfbbe91f-114b-4e33-a356-511602c939e7","Type":"ContainerStarted","Data":"842226819ac8e15423a900c4564e7371e5627259c62315c58de0ab975112694a"}
Nov 03 08:40:58.276267 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:58.276265    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6d8c96b47-jh99f" event={"ID":"bfbbe91f-114b-4e33-a356-511602c939e7","Type":"ContainerStarted","Data":"a010037349d71c7f2f7a0e7a107837c8656587ad99e811ef9ad3d17bc83ea46e"}
Nov 03 08:40:58.497426 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:58.497375    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6d8c96b47-jh99f" podStartSLOduration=1.396088356 podStartE2EDuration="3.497361255s" totalImagesPullingTime="2.101272899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:40:58.447030922 +0000 UTC m=+82.716094373" watchObservedRunningTime="2026-11-03 08:40:58.497361255 +0000 UTC m=+82.766424689"
Nov 03 08:40:58.912316 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:58.912283    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-68fbfd85cb-wbnwf"]
Nov 03 08:40:58.915507 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:58.915492    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68fbfd85cb-wbnwf"
Nov 03 08:40:58.955833 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:58.955802    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d066ebe-78bd-41e3-b17d-c39567f3efee-oauth-serving-cert\") pod \"console-68fbfd85cb-wbnwf\" (UID: \"1d066ebe-78bd-41e3-b17d-c39567f3efee\") " pod="openshift-console/console-68fbfd85cb-wbnwf"
Nov 03 08:40:58.955987 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:58.955840    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d066ebe-78bd-41e3-b17d-c39567f3efee-tmp\") pod \"console-68fbfd85cb-wbnwf\" (UID: \"1d066ebe-78bd-41e3-b17d-c39567f3efee\") " pod="openshift-console/console-68fbfd85cb-wbnwf"
Nov 03 08:40:58.955987 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:58.955862    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d066ebe-78bd-41e3-b17d-c39567f3efee-service-ca\") pod \"console-68fbfd85cb-wbnwf\" (UID: \"1d066ebe-78bd-41e3-b17d-c39567f3efee\") " pod="openshift-console/console-68fbfd85cb-wbnwf"
Nov 03 08:40:58.955987 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:58.955970    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d066ebe-78bd-41e3-b17d-c39567f3efee-trusted-ca-bundle\") pod \"console-68fbfd85cb-wbnwf\" (UID: \"1d066ebe-78bd-41e3-b17d-c39567f3efee\") " pod="openshift-console/console-68fbfd85cb-wbnwf"
Nov 03 08:40:58.956099 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:58.955988    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbbhp\" (UniqueName: \"kubernetes.io/projected/1d066ebe-78bd-41e3-b17d-c39567f3efee-kube-api-access-mbbhp\") pod \"console-68fbfd85cb-wbnwf\" (UID: \"1d066ebe-78bd-41e3-b17d-c39567f3efee\") " pod="openshift-console/console-68fbfd85cb-wbnwf"
Nov 03 08:40:58.956099 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:58.956021    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d066ebe-78bd-41e3-b17d-c39567f3efee-console-serving-cert\") pod \"console-68fbfd85cb-wbnwf\" (UID: \"1d066ebe-78bd-41e3-b17d-c39567f3efee\") " pod="openshift-console/console-68fbfd85cb-wbnwf"
Nov 03 08:40:58.956099 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:58.956064    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d066ebe-78bd-41e3-b17d-c39567f3efee-console-oauth-config\") pod \"console-68fbfd85cb-wbnwf\" (UID: \"1d066ebe-78bd-41e3-b17d-c39567f3efee\") " pod="openshift-console/console-68fbfd85cb-wbnwf"
Nov 03 08:40:58.956099 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:58.956095    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d066ebe-78bd-41e3-b17d-c39567f3efee-console-config\") pod \"console-68fbfd85cb-wbnwf\" (UID: \"1d066ebe-78bd-41e3-b17d-c39567f3efee\") " pod="openshift-console/console-68fbfd85cb-wbnwf"
Nov 03 08:40:59.011443 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:59.011415    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68fbfd85cb-wbnwf"]
Nov 03 08:40:59.056576 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:59.056548    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d066ebe-78bd-41e3-b17d-c39567f3efee-oauth-serving-cert\") pod \"console-68fbfd85cb-wbnwf\" (UID: \"1d066ebe-78bd-41e3-b17d-c39567f3efee\") " pod="openshift-console/console-68fbfd85cb-wbnwf"
Nov 03 08:40:59.056731 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:59.056585    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d066ebe-78bd-41e3-b17d-c39567f3efee-tmp\") pod \"console-68fbfd85cb-wbnwf\" (UID: \"1d066ebe-78bd-41e3-b17d-c39567f3efee\") " pod="openshift-console/console-68fbfd85cb-wbnwf"
Nov 03 08:40:59.056731 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:59.056604    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d066ebe-78bd-41e3-b17d-c39567f3efee-service-ca\") pod \"console-68fbfd85cb-wbnwf\" (UID: \"1d066ebe-78bd-41e3-b17d-c39567f3efee\") " pod="openshift-console/console-68fbfd85cb-wbnwf"
Nov 03 08:40:59.056731 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:59.056720    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d066ebe-78bd-41e3-b17d-c39567f3efee-trusted-ca-bundle\") pod \"console-68fbfd85cb-wbnwf\" (UID: \"1d066ebe-78bd-41e3-b17d-c39567f3efee\") " pod="openshift-console/console-68fbfd85cb-wbnwf"
Nov 03 08:40:59.056909 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:59.056755    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbbhp\" (UniqueName: \"kubernetes.io/projected/1d066ebe-78bd-41e3-b17d-c39567f3efee-kube-api-access-mbbhp\") pod \"console-68fbfd85cb-wbnwf\" (UID: \"1d066ebe-78bd-41e3-b17d-c39567f3efee\") " pod="openshift-console/console-68fbfd85cb-wbnwf"
Nov 03 08:40:59.056909 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:59.056798    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d066ebe-78bd-41e3-b17d-c39567f3efee-console-serving-cert\") pod \"console-68fbfd85cb-wbnwf\" (UID: \"1d066ebe-78bd-41e3-b17d-c39567f3efee\") " pod="openshift-console/console-68fbfd85cb-wbnwf"
Nov 03 08:40:59.056909 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:59.056830    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d066ebe-78bd-41e3-b17d-c39567f3efee-console-oauth-config\") pod \"console-68fbfd85cb-wbnwf\" (UID: \"1d066ebe-78bd-41e3-b17d-c39567f3efee\") " pod="openshift-console/console-68fbfd85cb-wbnwf"
Nov 03 08:40:59.056909 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:59.056848    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d066ebe-78bd-41e3-b17d-c39567f3efee-console-config\") pod \"console-68fbfd85cb-wbnwf\" (UID: \"1d066ebe-78bd-41e3-b17d-c39567f3efee\") " pod="openshift-console/console-68fbfd85cb-wbnwf"
Nov 03 08:40:59.057140 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:59.057015    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d066ebe-78bd-41e3-b17d-c39567f3efee-tmp\") pod \"console-68fbfd85cb-wbnwf\" (UID: \"1d066ebe-78bd-41e3-b17d-c39567f3efee\") " pod="openshift-console/console-68fbfd85cb-wbnwf"
Nov 03 08:40:59.057453 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:59.057428    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d066ebe-78bd-41e3-b17d-c39567f3efee-service-ca\") pod \"console-68fbfd85cb-wbnwf\" (UID: \"1d066ebe-78bd-41e3-b17d-c39567f3efee\") " pod="openshift-console/console-68fbfd85cb-wbnwf"
Nov 03 08:40:59.057767 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:59.057650    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d066ebe-78bd-41e3-b17d-c39567f3efee-oauth-serving-cert\") pod \"console-68fbfd85cb-wbnwf\" (UID: \"1d066ebe-78bd-41e3-b17d-c39567f3efee\") " pod="openshift-console/console-68fbfd85cb-wbnwf"
Nov 03 08:40:59.057870 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:59.057837    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d066ebe-78bd-41e3-b17d-c39567f3efee-console-config\") pod \"console-68fbfd85cb-wbnwf\" (UID: \"1d066ebe-78bd-41e3-b17d-c39567f3efee\") " pod="openshift-console/console-68fbfd85cb-wbnwf"
Nov 03 08:40:59.057870 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:59.057844    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d066ebe-78bd-41e3-b17d-c39567f3efee-trusted-ca-bundle\") pod \"console-68fbfd85cb-wbnwf\" (UID: \"1d066ebe-78bd-41e3-b17d-c39567f3efee\") " pod="openshift-console/console-68fbfd85cb-wbnwf"
Nov 03 08:40:59.059456 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:59.059436    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d066ebe-78bd-41e3-b17d-c39567f3efee-console-oauth-config\") pod \"console-68fbfd85cb-wbnwf\" (UID: \"1d066ebe-78bd-41e3-b17d-c39567f3efee\") " pod="openshift-console/console-68fbfd85cb-wbnwf"
Nov 03 08:40:59.059594 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:59.059575    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d066ebe-78bd-41e3-b17d-c39567f3efee-console-serving-cert\") pod \"console-68fbfd85cb-wbnwf\" (UID: \"1d066ebe-78bd-41e3-b17d-c39567f3efee\") " pod="openshift-console/console-68fbfd85cb-wbnwf"
Nov 03 08:40:59.116350 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:59.116321    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbbhp\" (UniqueName: \"kubernetes.io/projected/1d066ebe-78bd-41e3-b17d-c39567f3efee-kube-api-access-mbbhp\") pod \"console-68fbfd85cb-wbnwf\" (UID: \"1d066ebe-78bd-41e3-b17d-c39567f3efee\") " pod="openshift-console/console-68fbfd85cb-wbnwf"
Nov 03 08:40:59.226752 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:59.226678    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68fbfd85cb-wbnwf"
Nov 03 08:40:59.598715 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:40:59.598692    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68fbfd85cb-wbnwf"]
Nov 03 08:40:59.601158 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:40:59.601123    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d066ebe_78bd_41e3_b17d_c39567f3efee.slice/crio-62d72bf831d8e8c8012f1cab6f62b5bf48f3ddcb22d115ffd30148d3007e6a44 WatchSource:0}: Error finding container 62d72bf831d8e8c8012f1cab6f62b5bf48f3ddcb22d115ffd30148d3007e6a44: Status 404 returned error can't find the container with id 62d72bf831d8e8c8012f1cab6f62b5bf48f3ddcb22d115ffd30148d3007e6a44
Nov 03 08:41:00.286281 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:00.286246    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68fbfd85cb-wbnwf" event={"ID":"1d066ebe-78bd-41e3-b17d-c39567f3efee","Type":"ContainerStarted","Data":"8f10e210682d58252fb0b2b14f399c34416278e721fd19b5fdb9d5dc1dfbdf07"}
Nov 03 08:41:00.286451 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:00.286305    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68fbfd85cb-wbnwf" event={"ID":"1d066ebe-78bd-41e3-b17d-c39567f3efee","Type":"ContainerStarted","Data":"62d72bf831d8e8c8012f1cab6f62b5bf48f3ddcb22d115ffd30148d3007e6a44"}
Nov 03 08:41:00.405303 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:00.405254    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-68fbfd85cb-wbnwf" podStartSLOduration=2.405240624 podStartE2EDuration="2.405240624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:41:00.404645227 +0000 UTC m=+84.673708684" watchObservedRunningTime="2026-11-03 08:41:00.405240624 +0000 UTC m=+84.674304110"
Nov 03 08:41:04.353885 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:04.353829    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-65dff66875-qcbkf" podUID="aa0e821d-8b15-414d-b62e-7a311590e1d2" containerName="console" containerID="cri-o://f4057991ee1ace24cd2a39e44530fee09ca3333c27dc74244ea242aa607cf231" gracePeriod=15
Nov 03 08:41:04.605873 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:04.605819    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65dff66875-qcbkf"
Nov 03 08:41:04.702818 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:04.702788    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa0e821d-8b15-414d-b62e-7a311590e1d2-tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa0e821d-8b15-414d-b62e-7a311590e1d2-tmp\") pod \"aa0e821d-8b15-414d-b62e-7a311590e1d2\" (UID: \"aa0e821d-8b15-414d-b62e-7a311590e1d2\") "
Nov 03 08:41:04.702976 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:04.702836    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa0e821d-8b15-414d-b62e-7a311590e1d2-kube-api-access-fw87b\" (UniqueName: \"kubernetes.io/projected/aa0e821d-8b15-414d-b62e-7a311590e1d2-kube-api-access-fw87b\") pod \"aa0e821d-8b15-414d-b62e-7a311590e1d2\" (UID: \"aa0e821d-8b15-414d-b62e-7a311590e1d2\") "
Nov 03 08:41:04.702976 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:04.702864    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa0e821d-8b15-414d-b62e-7a311590e1d2-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa0e821d-8b15-414d-b62e-7a311590e1d2-console-oauth-config\") pod \"aa0e821d-8b15-414d-b62e-7a311590e1d2\" (UID: \"aa0e821d-8b15-414d-b62e-7a311590e1d2\") "
Nov 03 08:41:04.702976 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:04.702907    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa0e821d-8b15-414d-b62e-7a311590e1d2-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa0e821d-8b15-414d-b62e-7a311590e1d2-console-serving-cert\") pod \"aa0e821d-8b15-414d-b62e-7a311590e1d2\" (UID: \"aa0e821d-8b15-414d-b62e-7a311590e1d2\") "
Nov 03 08:41:04.702976 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:04.702968    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa0e821d-8b15-414d-b62e-7a311590e1d2-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa0e821d-8b15-414d-b62e-7a311590e1d2-oauth-serving-cert\") pod \"aa0e821d-8b15-414d-b62e-7a311590e1d2\" (UID: \"aa0e821d-8b15-414d-b62e-7a311590e1d2\") "
Nov 03 08:41:04.703219 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:04.703009    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa0e821d-8b15-414d-b62e-7a311590e1d2-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa0e821d-8b15-414d-b62e-7a311590e1d2-service-ca\") pod \"aa0e821d-8b15-414d-b62e-7a311590e1d2\" (UID: \"aa0e821d-8b15-414d-b62e-7a311590e1d2\") "
Nov 03 08:41:04.703219 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:04.703060    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa0e821d-8b15-414d-b62e-7a311590e1d2-console-config\" (UniqueName: \"kubernetes.io/configmap/aa0e821d-8b15-414d-b62e-7a311590e1d2-console-config\") pod \"aa0e821d-8b15-414d-b62e-7a311590e1d2\" (UID: \"aa0e821d-8b15-414d-b62e-7a311590e1d2\") "
Nov 03 08:41:04.703219 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:04.703123    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa0e821d-8b15-414d-b62e-7a311590e1d2-tmp" pod "aa0e821d-8b15-414d-b62e-7a311590e1d2" (UID: "aa0e821d-8b15-414d-b62e-7a311590e1d2"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:41:04.703369 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:04.703291    4473 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa0e821d-8b15-414d-b62e-7a311590e1d2-tmp\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:41:04.703469 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:04.703442    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa0e821d-8b15-414d-b62e-7a311590e1d2-oauth-serving-cert" pod "aa0e821d-8b15-414d-b62e-7a311590e1d2" (UID: "aa0e821d-8b15-414d-b62e-7a311590e1d2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:41:04.703526 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:04.703449    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa0e821d-8b15-414d-b62e-7a311590e1d2-service-ca" pod "aa0e821d-8b15-414d-b62e-7a311590e1d2" (UID: "aa0e821d-8b15-414d-b62e-7a311590e1d2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:41:04.703526 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:04.703469    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa0e821d-8b15-414d-b62e-7a311590e1d2-console-config" pod "aa0e821d-8b15-414d-b62e-7a311590e1d2" (UID: "aa0e821d-8b15-414d-b62e-7a311590e1d2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:41:04.705424 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:04.705396    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa0e821d-8b15-414d-b62e-7a311590e1d2-console-serving-cert" pod "aa0e821d-8b15-414d-b62e-7a311590e1d2" (UID: "aa0e821d-8b15-414d-b62e-7a311590e1d2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:41:04.705424 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:04.705414    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa0e821d-8b15-414d-b62e-7a311590e1d2-console-oauth-config" pod "aa0e821d-8b15-414d-b62e-7a311590e1d2" (UID: "aa0e821d-8b15-414d-b62e-7a311590e1d2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:41:04.705557 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:04.705444    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa0e821d-8b15-414d-b62e-7a311590e1d2-kube-api-access-fw87b" pod "aa0e821d-8b15-414d-b62e-7a311590e1d2" (UID: "aa0e821d-8b15-414d-b62e-7a311590e1d2"). InnerVolumeSpecName "kube-api-access-fw87b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:41:04.804467 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:04.804439    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw87b\" (UniqueName: \"kubernetes.io/projected/aa0e821d-8b15-414d-b62e-7a311590e1d2-kube-api-access-fw87b\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:41:04.804467 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:04.804465    4473 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa0e821d-8b15-414d-b62e-7a311590e1d2-console-oauth-config\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:41:04.804644 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:04.804478    4473 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa0e821d-8b15-414d-b62e-7a311590e1d2-console-serving-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:41:04.804644 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:04.804486    4473 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa0e821d-8b15-414d-b62e-7a311590e1d2-oauth-serving-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:41:04.804644 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:04.804495    4473 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa0e821d-8b15-414d-b62e-7a311590e1d2-service-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:41:04.804644 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:04.804502    4473 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa0e821d-8b15-414d-b62e-7a311590e1d2-console-config\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:41:05.306126 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:05.306087    4473 generic.go:356] "Generic (PLEG): container finished" podID="aa0e821d-8b15-414d-b62e-7a311590e1d2" containerID="f4057991ee1ace24cd2a39e44530fee09ca3333c27dc74244ea242aa607cf231" exitCode=0
Nov 03 08:41:05.306306 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:05.306134    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65dff66875-qcbkf"
Nov 03 08:41:05.306306 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:05.306174    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65dff66875-qcbkf" event={"ID":"aa0e821d-8b15-414d-b62e-7a311590e1d2","Type":"ContainerDied","Data":"f4057991ee1ace24cd2a39e44530fee09ca3333c27dc74244ea242aa607cf231"}
Nov 03 08:41:05.306306 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:05.306221    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65dff66875-qcbkf" event={"ID":"aa0e821d-8b15-414d-b62e-7a311590e1d2","Type":"ContainerDied","Data":"aa1784b440a7fc755075d35d30844daf0a2caf41e0365605dfc6bfbe06e5a34e"}
Nov 03 08:41:05.306306 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:05.306241    4473 scope.go:122] "RemoveContainer" containerID="f4057991ee1ace24cd2a39e44530fee09ca3333c27dc74244ea242aa607cf231"
Nov 03 08:41:05.315172 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:05.315155    4473 scope.go:122] "RemoveContainer" containerID="f4057991ee1ace24cd2a39e44530fee09ca3333c27dc74244ea242aa607cf231"
Nov 03 08:41:05.315436 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:41:05.315416    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4057991ee1ace24cd2a39e44530fee09ca3333c27dc74244ea242aa607cf231\": container with ID starting with f4057991ee1ace24cd2a39e44530fee09ca3333c27dc74244ea242aa607cf231 not found: ID does not exist" containerID="f4057991ee1ace24cd2a39e44530fee09ca3333c27dc74244ea242aa607cf231"
Nov 03 08:41:05.315499 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:05.315447    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4057991ee1ace24cd2a39e44530fee09ca3333c27dc74244ea242aa607cf231"} err="failed to get container status \"f4057991ee1ace24cd2a39e44530fee09ca3333c27dc74244ea242aa607cf231\": rpc error: code = NotFound desc = could not find container \"f4057991ee1ace24cd2a39e44530fee09ca3333c27dc74244ea242aa607cf231\": container with ID starting with f4057991ee1ace24cd2a39e44530fee09ca3333c27dc74244ea242aa607cf231 not found: ID does not exist"
Nov 03 08:41:05.475298 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:05.475268    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65dff66875-qcbkf"]
Nov 03 08:41:05.525355 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:05.525325    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65dff66875-qcbkf"]
Nov 03 08:41:05.871157 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:05.871128    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa0e821d-8b15-414d-b62e-7a311590e1d2" path="/var/lib/kubelet/pods/aa0e821d-8b15-414d-b62e-7a311590e1d2/volumes"
Nov 03 08:41:09.227331 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:09.227289    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68fbfd85cb-wbnwf"
Nov 03 08:41:09.227886 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:09.227359    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68fbfd85cb-wbnwf"
Nov 03 08:41:09.231860 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:09.231838    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68fbfd85cb-wbnwf"
Nov 03 08:41:09.325281 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:09.325258    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68fbfd85cb-wbnwf"
Nov 03 08:41:09.517803 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:09.517730    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c9db68cb9-qq24h"]
Nov 03 08:41:16.936127 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:41:16.936093    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:41:34.537420 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:34.537354    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6c9db68cb9-qq24h" podUID="bb543947-8e68-4d9e-a352-f546e5249358" containerName="console" containerID="cri-o://d3cf451241e268198a7009022f6025c7b913be09e6976844bd6e216879fddf8d" gracePeriod=15
Nov 03 08:41:34.772754 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:34.772731    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c9db68cb9-qq24h"
Nov 03 08:41:34.865455 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:34.865434    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb543947-8e68-4d9e-a352-f546e5249358-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb543947-8e68-4d9e-a352-f546e5249358-trusted-ca-bundle\") pod \"bb543947-8e68-4d9e-a352-f546e5249358\" (UID: \"bb543947-8e68-4d9e-a352-f546e5249358\") "
Nov 03 08:41:34.865595 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:34.865477    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb543947-8e68-4d9e-a352-f546e5249358-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bb543947-8e68-4d9e-a352-f546e5249358-console-serving-cert\") pod \"bb543947-8e68-4d9e-a352-f546e5249358\" (UID: \"bb543947-8e68-4d9e-a352-f546e5249358\") "
Nov 03 08:41:34.865595 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:34.865499    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb543947-8e68-4d9e-a352-f546e5249358-tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb543947-8e68-4d9e-a352-f546e5249358-tmp\") pod \"bb543947-8e68-4d9e-a352-f546e5249358\" (UID: \"bb543947-8e68-4d9e-a352-f546e5249358\") "
Nov 03 08:41:34.865595 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:34.865524    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb543947-8e68-4d9e-a352-f546e5249358-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bb543947-8e68-4d9e-a352-f546e5249358-console-oauth-config\") pod \"bb543947-8e68-4d9e-a352-f546e5249358\" (UID: \"bb543947-8e68-4d9e-a352-f546e5249358\") "
Nov 03 08:41:34.865595 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:34.865567    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb543947-8e68-4d9e-a352-f546e5249358-console-config\" (UniqueName: \"kubernetes.io/configmap/bb543947-8e68-4d9e-a352-f546e5249358-console-config\") pod \"bb543947-8e68-4d9e-a352-f546e5249358\" (UID: \"bb543947-8e68-4d9e-a352-f546e5249358\") "
Nov 03 08:41:34.865750 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:34.865596    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb543947-8e68-4d9e-a352-f546e5249358-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bb543947-8e68-4d9e-a352-f546e5249358-oauth-serving-cert\") pod \"bb543947-8e68-4d9e-a352-f546e5249358\" (UID: \"bb543947-8e68-4d9e-a352-f546e5249358\") "
Nov 03 08:41:34.865750 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:34.865633    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb543947-8e68-4d9e-a352-f546e5249358-kube-api-access-2mvwv\" (UniqueName: \"kubernetes.io/projected/bb543947-8e68-4d9e-a352-f546e5249358-kube-api-access-2mvwv\") pod \"bb543947-8e68-4d9e-a352-f546e5249358\" (UID: \"bb543947-8e68-4d9e-a352-f546e5249358\") "
Nov 03 08:41:34.865750 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:34.865674    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb543947-8e68-4d9e-a352-f546e5249358-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb543947-8e68-4d9e-a352-f546e5249358-service-ca\") pod \"bb543947-8e68-4d9e-a352-f546e5249358\" (UID: \"bb543947-8e68-4d9e-a352-f546e5249358\") "
Nov 03 08:41:34.865936 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:34.865904    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb543947-8e68-4d9e-a352-f546e5249358-tmp" pod "bb543947-8e68-4d9e-a352-f546e5249358" (UID: "bb543947-8e68-4d9e-a352-f546e5249358"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:41:34.866140 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:34.866006    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb543947-8e68-4d9e-a352-f546e5249358-trusted-ca-bundle" pod "bb543947-8e68-4d9e-a352-f546e5249358" (UID: "bb543947-8e68-4d9e-a352-f546e5249358"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:41:34.866140 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:34.866131    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb543947-8e68-4d9e-a352-f546e5249358-console-config" pod "bb543947-8e68-4d9e-a352-f546e5249358" (UID: "bb543947-8e68-4d9e-a352-f546e5249358"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:41:34.866262 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:34.866155    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb543947-8e68-4d9e-a352-f546e5249358-oauth-serving-cert" pod "bb543947-8e68-4d9e-a352-f546e5249358" (UID: "bb543947-8e68-4d9e-a352-f546e5249358"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:41:34.866262 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:34.866208    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb543947-8e68-4d9e-a352-f546e5249358-service-ca" pod "bb543947-8e68-4d9e-a352-f546e5249358" (UID: "bb543947-8e68-4d9e-a352-f546e5249358"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:41:34.867866 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:34.867845    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb543947-8e68-4d9e-a352-f546e5249358-kube-api-access-2mvwv" pod "bb543947-8e68-4d9e-a352-f546e5249358" (UID: "bb543947-8e68-4d9e-a352-f546e5249358"). InnerVolumeSpecName "kube-api-access-2mvwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:41:34.867932 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:34.867862    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb543947-8e68-4d9e-a352-f546e5249358-console-serving-cert" pod "bb543947-8e68-4d9e-a352-f546e5249358" (UID: "bb543947-8e68-4d9e-a352-f546e5249358"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:41:34.867932 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:34.867889    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb543947-8e68-4d9e-a352-f546e5249358-console-oauth-config" pod "bb543947-8e68-4d9e-a352-f546e5249358" (UID: "bb543947-8e68-4d9e-a352-f546e5249358"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:41:34.966844 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:34.966812    4473 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bb543947-8e68-4d9e-a352-f546e5249358-oauth-serving-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:41:34.966844 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:34.966836    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mvwv\" (UniqueName: \"kubernetes.io/projected/bb543947-8e68-4d9e-a352-f546e5249358-kube-api-access-2mvwv\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:41:34.966844 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:34.966848    4473 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bb543947-8e68-4d9e-a352-f546e5249358-service-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:41:34.967085 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:34.966857    4473 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb543947-8e68-4d9e-a352-f546e5249358-trusted-ca-bundle\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:41:34.967085 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:34.966866    4473 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bb543947-8e68-4d9e-a352-f546e5249358-console-serving-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:41:34.967085 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:34.966876    4473 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb543947-8e68-4d9e-a352-f546e5249358-tmp\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:41:34.967085 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:34.966884    4473 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bb543947-8e68-4d9e-a352-f546e5249358-console-oauth-config\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:41:34.967085 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:34.966892    4473 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bb543947-8e68-4d9e-a352-f546e5249358-console-config\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:41:35.421613 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:35.421581    4473 generic.go:356] "Generic (PLEG): container finished" podID="bb543947-8e68-4d9e-a352-f546e5249358" containerID="d3cf451241e268198a7009022f6025c7b913be09e6976844bd6e216879fddf8d" exitCode=0
Nov 03 08:41:35.421793 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:35.421636    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c9db68cb9-qq24h"
Nov 03 08:41:35.421793 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:35.421641    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c9db68cb9-qq24h" event={"ID":"bb543947-8e68-4d9e-a352-f546e5249358","Type":"ContainerDied","Data":"d3cf451241e268198a7009022f6025c7b913be09e6976844bd6e216879fddf8d"}
Nov 03 08:41:35.421793 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:35.421752    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c9db68cb9-qq24h" event={"ID":"bb543947-8e68-4d9e-a352-f546e5249358","Type":"ContainerDied","Data":"22f8fd714ff6f7bb22586e5669358e1b0ee3d18bc5ba2e6b291aea529551dbba"}
Nov 03 08:41:35.421793 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:35.421772    4473 scope.go:122] "RemoveContainer" containerID="d3cf451241e268198a7009022f6025c7b913be09e6976844bd6e216879fddf8d"
Nov 03 08:41:35.431753 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:35.431727    4473 scope.go:122] "RemoveContainer" containerID="d3cf451241e268198a7009022f6025c7b913be09e6976844bd6e216879fddf8d"
Nov 03 08:41:35.432077 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:41:35.432053    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3cf451241e268198a7009022f6025c7b913be09e6976844bd6e216879fddf8d\": container with ID starting with d3cf451241e268198a7009022f6025c7b913be09e6976844bd6e216879fddf8d not found: ID does not exist" containerID="d3cf451241e268198a7009022f6025c7b913be09e6976844bd6e216879fddf8d"
Nov 03 08:41:35.432153 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:35.432091    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3cf451241e268198a7009022f6025c7b913be09e6976844bd6e216879fddf8d"} err="failed to get container status \"d3cf451241e268198a7009022f6025c7b913be09e6976844bd6e216879fddf8d\": rpc error: code = NotFound desc = could not find container \"d3cf451241e268198a7009022f6025c7b913be09e6976844bd6e216879fddf8d\": container with ID starting with d3cf451241e268198a7009022f6025c7b913be09e6976844bd6e216879fddf8d not found: ID does not exist"
Nov 03 08:41:35.590404 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:35.590374    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c9db68cb9-qq24h"]
Nov 03 08:41:35.640879 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:35.640843    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c9db68cb9-qq24h"]
Nov 03 08:41:35.867017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:41:35.866988    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb543947-8e68-4d9e-a352-f546e5249358" path="/var/lib/kubelet/pods/bb543947-8e68-4d9e-a352-f546e5249358/volumes"
Nov 03 08:41:46.936557 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:41:46.936471    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:42:10.349883 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:10.349854    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bf4b887bc-p5l6b"]
Nov 03 08:42:10.350385 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:10.350177    4473 cpu_manager.go:414] "containerMap: removing container" podUID="aa0e821d-8b15-414d-b62e-7a311590e1d2" containerName="console"
Nov 03 08:42:10.350385 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:10.350190    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa0e821d-8b15-414d-b62e-7a311590e1d2" containerName="console"
Nov 03 08:42:10.350385 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:10.350204    4473 cpu_manager.go:414] "containerMap: removing container" podUID="bb543947-8e68-4d9e-a352-f546e5249358" containerName="console"
Nov 03 08:42:10.350385 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:10.350210    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb543947-8e68-4d9e-a352-f546e5249358" containerName="console"
Nov 03 08:42:10.350385 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:10.350264    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa0e821d-8b15-414d-b62e-7a311590e1d2" containerName="console"
Nov 03 08:42:10.350385 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:10.350272    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb543947-8e68-4d9e-a352-f546e5249358" containerName="console"
Nov 03 08:42:10.354171 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:10.354155    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bf4b887bc-p5l6b"
Nov 03 08:42:10.413221 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:10.413186    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bf4b887bc-p5l6b"]
Nov 03 08:42:10.450128 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:10.450092    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a18a1e24-c5b9-4328-952a-e235d0249337-console-serving-cert\") pod \"console-5bf4b887bc-p5l6b\" (UID: \"a18a1e24-c5b9-4328-952a-e235d0249337\") " pod="openshift-console/console-5bf4b887bc-p5l6b"
Nov 03 08:42:10.450128 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:10.450129    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a18a1e24-c5b9-4328-952a-e235d0249337-service-ca\") pod \"console-5bf4b887bc-p5l6b\" (UID: \"a18a1e24-c5b9-4328-952a-e235d0249337\") " pod="openshift-console/console-5bf4b887bc-p5l6b"
Nov 03 08:42:10.450311 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:10.450153    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a18a1e24-c5b9-4328-952a-e235d0249337-trusted-ca-bundle\") pod \"console-5bf4b887bc-p5l6b\" (UID: \"a18a1e24-c5b9-4328-952a-e235d0249337\") " pod="openshift-console/console-5bf4b887bc-p5l6b"
Nov 03 08:42:10.450311 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:10.450231    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a18a1e24-c5b9-4328-952a-e235d0249337-console-oauth-config\") pod \"console-5bf4b887bc-p5l6b\" (UID: \"a18a1e24-c5b9-4328-952a-e235d0249337\") " pod="openshift-console/console-5bf4b887bc-p5l6b"
Nov 03 08:42:10.450311 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:10.450255    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a18a1e24-c5b9-4328-952a-e235d0249337-oauth-serving-cert\") pod \"console-5bf4b887bc-p5l6b\" (UID: \"a18a1e24-c5b9-4328-952a-e235d0249337\") " pod="openshift-console/console-5bf4b887bc-p5l6b"
Nov 03 08:42:10.450311 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:10.450294    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a18a1e24-c5b9-4328-952a-e235d0249337-console-config\") pod \"console-5bf4b887bc-p5l6b\" (UID: \"a18a1e24-c5b9-4328-952a-e235d0249337\") " pod="openshift-console/console-5bf4b887bc-p5l6b"
Nov 03 08:42:10.450461 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:10.450356    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lnjc\" (UniqueName: \"kubernetes.io/projected/a18a1e24-c5b9-4328-952a-e235d0249337-kube-api-access-5lnjc\") pod \"console-5bf4b887bc-p5l6b\" (UID: \"a18a1e24-c5b9-4328-952a-e235d0249337\") " pod="openshift-console/console-5bf4b887bc-p5l6b"
Nov 03 08:42:10.450501 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:10.450460    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a18a1e24-c5b9-4328-952a-e235d0249337-tmp\") pod \"console-5bf4b887bc-p5l6b\" (UID: \"a18a1e24-c5b9-4328-952a-e235d0249337\") " pod="openshift-console/console-5bf4b887bc-p5l6b"
Nov 03 08:42:10.551814 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:10.551785    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lnjc\" (UniqueName: \"kubernetes.io/projected/a18a1e24-c5b9-4328-952a-e235d0249337-kube-api-access-5lnjc\") pod \"console-5bf4b887bc-p5l6b\" (UID: \"a18a1e24-c5b9-4328-952a-e235d0249337\") " pod="openshift-console/console-5bf4b887bc-p5l6b"
Nov 03 08:42:10.552031 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:10.551833    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a18a1e24-c5b9-4328-952a-e235d0249337-tmp\") pod \"console-5bf4b887bc-p5l6b\" (UID: \"a18a1e24-c5b9-4328-952a-e235d0249337\") " pod="openshift-console/console-5bf4b887bc-p5l6b"
Nov 03 08:42:10.552031 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:10.551858    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a18a1e24-c5b9-4328-952a-e235d0249337-console-serving-cert\") pod \"console-5bf4b887bc-p5l6b\" (UID: \"a18a1e24-c5b9-4328-952a-e235d0249337\") " pod="openshift-console/console-5bf4b887bc-p5l6b"
Nov 03 08:42:10.552031 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:10.551884    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a18a1e24-c5b9-4328-952a-e235d0249337-service-ca\") pod \"console-5bf4b887bc-p5l6b\" (UID: \"a18a1e24-c5b9-4328-952a-e235d0249337\") " pod="openshift-console/console-5bf4b887bc-p5l6b"
Nov 03 08:42:10.552031 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:10.551900    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a18a1e24-c5b9-4328-952a-e235d0249337-trusted-ca-bundle\") pod \"console-5bf4b887bc-p5l6b\" (UID: \"a18a1e24-c5b9-4328-952a-e235d0249337\") " pod="openshift-console/console-5bf4b887bc-p5l6b"
Nov 03 08:42:10.552031 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:10.551919    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a18a1e24-c5b9-4328-952a-e235d0249337-console-oauth-config\") pod \"console-5bf4b887bc-p5l6b\" (UID: \"a18a1e24-c5b9-4328-952a-e235d0249337\") " pod="openshift-console/console-5bf4b887bc-p5l6b"
Nov 03 08:42:10.552031 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:10.551940    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a18a1e24-c5b9-4328-952a-e235d0249337-oauth-serving-cert\") pod \"console-5bf4b887bc-p5l6b\" (UID: \"a18a1e24-c5b9-4328-952a-e235d0249337\") " pod="openshift-console/console-5bf4b887bc-p5l6b"
Nov 03 08:42:10.552031 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:10.552016    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a18a1e24-c5b9-4328-952a-e235d0249337-console-config\") pod \"console-5bf4b887bc-p5l6b\" (UID: \"a18a1e24-c5b9-4328-952a-e235d0249337\") " pod="openshift-console/console-5bf4b887bc-p5l6b"
Nov 03 08:42:10.552353 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:10.552277    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a18a1e24-c5b9-4328-952a-e235d0249337-tmp\") pod \"console-5bf4b887bc-p5l6b\" (UID: \"a18a1e24-c5b9-4328-952a-e235d0249337\") " pod="openshift-console/console-5bf4b887bc-p5l6b"
Nov 03 08:42:10.552698 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:10.552672    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a18a1e24-c5b9-4328-952a-e235d0249337-service-ca\") pod \"console-5bf4b887bc-p5l6b\" (UID: \"a18a1e24-c5b9-4328-952a-e235d0249337\") " pod="openshift-console/console-5bf4b887bc-p5l6b"
Nov 03 08:42:10.552841 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:10.552749    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a18a1e24-c5b9-4328-952a-e235d0249337-oauth-serving-cert\") pod \"console-5bf4b887bc-p5l6b\" (UID: \"a18a1e24-c5b9-4328-952a-e235d0249337\") " pod="openshift-console/console-5bf4b887bc-p5l6b"
Nov 03 08:42:10.552920 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:10.552767    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a18a1e24-c5b9-4328-952a-e235d0249337-console-config\") pod \"console-5bf4b887bc-p5l6b\" (UID: \"a18a1e24-c5b9-4328-952a-e235d0249337\") " pod="openshift-console/console-5bf4b887bc-p5l6b"
Nov 03 08:42:10.552920 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:10.552864    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a18a1e24-c5b9-4328-952a-e235d0249337-trusted-ca-bundle\") pod \"console-5bf4b887bc-p5l6b\" (UID: \"a18a1e24-c5b9-4328-952a-e235d0249337\") " pod="openshift-console/console-5bf4b887bc-p5l6b"
Nov 03 08:42:10.554533 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:10.554514    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a18a1e24-c5b9-4328-952a-e235d0249337-console-oauth-config\") pod \"console-5bf4b887bc-p5l6b\" (UID: \"a18a1e24-c5b9-4328-952a-e235d0249337\") " pod="openshift-console/console-5bf4b887bc-p5l6b"
Nov 03 08:42:10.554640 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:10.554623    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a18a1e24-c5b9-4328-952a-e235d0249337-console-serving-cert\") pod \"console-5bf4b887bc-p5l6b\" (UID: \"a18a1e24-c5b9-4328-952a-e235d0249337\") " pod="openshift-console/console-5bf4b887bc-p5l6b"
Nov 03 08:42:10.612272 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:10.612209    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lnjc\" (UniqueName: \"kubernetes.io/projected/a18a1e24-c5b9-4328-952a-e235d0249337-kube-api-access-5lnjc\") pod \"console-5bf4b887bc-p5l6b\" (UID: \"a18a1e24-c5b9-4328-952a-e235d0249337\") " pod="openshift-console/console-5bf4b887bc-p5l6b"
Nov 03 08:42:10.668374 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:10.668341    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bf4b887bc-p5l6b"
Nov 03 08:42:11.039300 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:11.039277    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bf4b887bc-p5l6b"]
Nov 03 08:42:11.041322 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:42:11.041295    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda18a1e24_c5b9_4328_952a_e235d0249337.slice/crio-906d39337ba8b486e5f0b65e34d876a866bbc1893638bf0aeaea541c6a06261e WatchSource:0}: Error finding container 906d39337ba8b486e5f0b65e34d876a866bbc1893638bf0aeaea541c6a06261e: Status 404 returned error can't find the container with id 906d39337ba8b486e5f0b65e34d876a866bbc1893638bf0aeaea541c6a06261e
Nov 03 08:42:11.546869 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:11.546831    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bf4b887bc-p5l6b" event={"ID":"a18a1e24-c5b9-4328-952a-e235d0249337","Type":"ContainerStarted","Data":"a6c9febe777cb6353918c1ffd70fca52e556c5766c8ccd9f2550555b3fd509c3"}
Nov 03 08:42:11.547281 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:11.546896    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bf4b887bc-p5l6b" event={"ID":"a18a1e24-c5b9-4328-952a-e235d0249337","Type":"ContainerStarted","Data":"906d39337ba8b486e5f0b65e34d876a866bbc1893638bf0aeaea541c6a06261e"}
Nov 03 08:42:16.935844 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:42:16.935809    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:42:20.668453 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:20.668418    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5bf4b887bc-p5l6b"
Nov 03 08:42:20.668870 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:20.668478    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bf4b887bc-p5l6b"
Nov 03 08:42:20.673350 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:20.673330    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5bf4b887bc-p5l6b"
Nov 03 08:42:20.792274 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:20.792229    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5bf4b887bc-p5l6b" podStartSLOduration=10.792216767 podStartE2EDuration="10.792216767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:42:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:42:11.719359727 +0000 UTC m=+155.988423181" watchObservedRunningTime="2026-11-03 08:42:20.792216767 +0000 UTC m=+165.061280201"
Nov 03 08:42:21.586476 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:21.586446    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5bf4b887bc-p5l6b"
Nov 03 08:42:21.757054 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:21.757023    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68fbfd85cb-wbnwf"]
Nov 03 08:42:46.776348 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:46.776298    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-68fbfd85cb-wbnwf" podUID="1d066ebe-78bd-41e3-b17d-c39567f3efee" containerName="console" containerID="cri-o://8f10e210682d58252fb0b2b14f399c34416278e721fd19b5fdb9d5dc1dfbdf07" gracePeriod=15
Nov 03 08:42:46.936170 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:42:46.936134    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:42:47.015503 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:47.015477    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68fbfd85cb-wbnwf"
Nov 03 08:42:47.144611 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:47.144586    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d066ebe-78bd-41e3-b17d-c39567f3efee-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d066ebe-78bd-41e3-b17d-c39567f3efee-service-ca\") pod \"1d066ebe-78bd-41e3-b17d-c39567f3efee\" (UID: \"1d066ebe-78bd-41e3-b17d-c39567f3efee\") "
Nov 03 08:42:47.144777 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:47.144640    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d066ebe-78bd-41e3-b17d-c39567f3efee-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d066ebe-78bd-41e3-b17d-c39567f3efee-console-oauth-config\") pod \"1d066ebe-78bd-41e3-b17d-c39567f3efee\" (UID: \"1d066ebe-78bd-41e3-b17d-c39567f3efee\") "
Nov 03 08:42:47.144777 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:47.144672    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d066ebe-78bd-41e3-b17d-c39567f3efee-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d066ebe-78bd-41e3-b17d-c39567f3efee-console-serving-cert\") pod \"1d066ebe-78bd-41e3-b17d-c39567f3efee\" (UID: \"1d066ebe-78bd-41e3-b17d-c39567f3efee\") "
Nov 03 08:42:47.144777 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:47.144693    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d066ebe-78bd-41e3-b17d-c39567f3efee-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d066ebe-78bd-41e3-b17d-c39567f3efee-trusted-ca-bundle\") pod \"1d066ebe-78bd-41e3-b17d-c39567f3efee\" (UID: \"1d066ebe-78bd-41e3-b17d-c39567f3efee\") "
Nov 03 08:42:47.144777 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:47.144748    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d066ebe-78bd-41e3-b17d-c39567f3efee-kube-api-access-mbbhp\" (UniqueName: \"kubernetes.io/projected/1d066ebe-78bd-41e3-b17d-c39567f3efee-kube-api-access-mbbhp\") pod \"1d066ebe-78bd-41e3-b17d-c39567f3efee\" (UID: \"1d066ebe-78bd-41e3-b17d-c39567f3efee\") "
Nov 03 08:42:47.145016 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:47.144884    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d066ebe-78bd-41e3-b17d-c39567f3efee-console-config\" (UniqueName: \"kubernetes.io/configmap/1d066ebe-78bd-41e3-b17d-c39567f3efee-console-config\") pod \"1d066ebe-78bd-41e3-b17d-c39567f3efee\" (UID: \"1d066ebe-78bd-41e3-b17d-c39567f3efee\") "
Nov 03 08:42:47.145016 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:47.144957    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d066ebe-78bd-41e3-b17d-c39567f3efee-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d066ebe-78bd-41e3-b17d-c39567f3efee-oauth-serving-cert\") pod \"1d066ebe-78bd-41e3-b17d-c39567f3efee\" (UID: \"1d066ebe-78bd-41e3-b17d-c39567f3efee\") "
Nov 03 08:42:47.145016 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:47.145000    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d066ebe-78bd-41e3-b17d-c39567f3efee-tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d066ebe-78bd-41e3-b17d-c39567f3efee-tmp\") pod \"1d066ebe-78bd-41e3-b17d-c39567f3efee\" (UID: \"1d066ebe-78bd-41e3-b17d-c39567f3efee\") "
Nov 03 08:42:47.145255 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:47.145053    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d066ebe-78bd-41e3-b17d-c39567f3efee-service-ca" pod "1d066ebe-78bd-41e3-b17d-c39567f3efee" (UID: "1d066ebe-78bd-41e3-b17d-c39567f3efee"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:42:47.145635 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:47.145458    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d066ebe-78bd-41e3-b17d-c39567f3efee-trusted-ca-bundle" pod "1d066ebe-78bd-41e3-b17d-c39567f3efee" (UID: "1d066ebe-78bd-41e3-b17d-c39567f3efee"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:42:47.145635 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:47.145559    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d066ebe-78bd-41e3-b17d-c39567f3efee-tmp" pod "1d066ebe-78bd-41e3-b17d-c39567f3efee" (UID: "1d066ebe-78bd-41e3-b17d-c39567f3efee"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:42:47.145635 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:47.145516    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d066ebe-78bd-41e3-b17d-c39567f3efee-console-config" pod "1d066ebe-78bd-41e3-b17d-c39567f3efee" (UID: "1d066ebe-78bd-41e3-b17d-c39567f3efee"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:42:47.145831 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:47.145807    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d066ebe-78bd-41e3-b17d-c39567f3efee-oauth-serving-cert" pod "1d066ebe-78bd-41e3-b17d-c39567f3efee" (UID: "1d066ebe-78bd-41e3-b17d-c39567f3efee"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:42:47.147356 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:47.147324    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d066ebe-78bd-41e3-b17d-c39567f3efee-kube-api-access-mbbhp" pod "1d066ebe-78bd-41e3-b17d-c39567f3efee" (UID: "1d066ebe-78bd-41e3-b17d-c39567f3efee"). InnerVolumeSpecName "kube-api-access-mbbhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:42:47.147708 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:47.147682    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d066ebe-78bd-41e3-b17d-c39567f3efee-console-serving-cert" pod "1d066ebe-78bd-41e3-b17d-c39567f3efee" (UID: "1d066ebe-78bd-41e3-b17d-c39567f3efee"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:42:47.147708 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:47.147696    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d066ebe-78bd-41e3-b17d-c39567f3efee-console-oauth-config" pod "1d066ebe-78bd-41e3-b17d-c39567f3efee" (UID: "1d066ebe-78bd-41e3-b17d-c39567f3efee"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:42:47.245922 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:47.245892    4473 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d066ebe-78bd-41e3-b17d-c39567f3efee-console-serving-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:42:47.245922 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:47.245920    4473 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d066ebe-78bd-41e3-b17d-c39567f3efee-trusted-ca-bundle\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:42:47.245922 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:47.245930    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbbhp\" (UniqueName: \"kubernetes.io/projected/1d066ebe-78bd-41e3-b17d-c39567f3efee-kube-api-access-mbbhp\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:42:47.246150 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:47.245958    4473 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d066ebe-78bd-41e3-b17d-c39567f3efee-console-config\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:42:47.246150 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:47.245968    4473 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d066ebe-78bd-41e3-b17d-c39567f3efee-oauth-serving-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:42:47.246150 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:47.245976    4473 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d066ebe-78bd-41e3-b17d-c39567f3efee-tmp\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:42:47.246150 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:47.245985    4473 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d066ebe-78bd-41e3-b17d-c39567f3efee-service-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:42:47.246150 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:47.245993    4473 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d066ebe-78bd-41e3-b17d-c39567f3efee-console-oauth-config\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:42:47.674146 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:47.674116    4473 generic.go:356] "Generic (PLEG): container finished" podID="1d066ebe-78bd-41e3-b17d-c39567f3efee" containerID="8f10e210682d58252fb0b2b14f399c34416278e721fd19b5fdb9d5dc1dfbdf07" exitCode=0
Nov 03 08:42:47.674312 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:47.674170    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68fbfd85cb-wbnwf"
Nov 03 08:42:47.674312 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:47.674177    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68fbfd85cb-wbnwf" event={"ID":"1d066ebe-78bd-41e3-b17d-c39567f3efee","Type":"ContainerDied","Data":"8f10e210682d58252fb0b2b14f399c34416278e721fd19b5fdb9d5dc1dfbdf07"}
Nov 03 08:42:47.674312 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:47.674227    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68fbfd85cb-wbnwf" event={"ID":"1d066ebe-78bd-41e3-b17d-c39567f3efee","Type":"ContainerDied","Data":"62d72bf831d8e8c8012f1cab6f62b5bf48f3ddcb22d115ffd30148d3007e6a44"}
Nov 03 08:42:47.674312 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:47.674253    4473 scope.go:122] "RemoveContainer" containerID="8f10e210682d58252fb0b2b14f399c34416278e721fd19b5fdb9d5dc1dfbdf07"
Nov 03 08:42:47.682842 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:47.682825    4473 scope.go:122] "RemoveContainer" containerID="8f10e210682d58252fb0b2b14f399c34416278e721fd19b5fdb9d5dc1dfbdf07"
Nov 03 08:42:47.683147 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:42:47.683128    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f10e210682d58252fb0b2b14f399c34416278e721fd19b5fdb9d5dc1dfbdf07\": container with ID starting with 8f10e210682d58252fb0b2b14f399c34416278e721fd19b5fdb9d5dc1dfbdf07 not found: ID does not exist" containerID="8f10e210682d58252fb0b2b14f399c34416278e721fd19b5fdb9d5dc1dfbdf07"
Nov 03 08:42:47.683210 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:47.683159    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f10e210682d58252fb0b2b14f399c34416278e721fd19b5fdb9d5dc1dfbdf07"} err="failed to get container status \"8f10e210682d58252fb0b2b14f399c34416278e721fd19b5fdb9d5dc1dfbdf07\": rpc error: code = NotFound desc = could not find container \"8f10e210682d58252fb0b2b14f399c34416278e721fd19b5fdb9d5dc1dfbdf07\": container with ID starting with 8f10e210682d58252fb0b2b14f399c34416278e721fd19b5fdb9d5dc1dfbdf07 not found: ID does not exist"
Nov 03 08:42:47.845472 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:47.845446    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68fbfd85cb-wbnwf"]
Nov 03 08:42:47.894874 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:47.894841    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68fbfd85cb-wbnwf"]
Nov 03 08:42:49.867103 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:42:49.867068    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d066ebe-78bd-41e3-b17d-c39567f3efee" path="/var/lib/kubelet/pods/1d066ebe-78bd-41e3-b17d-c39567f3efee/volumes"
Nov 03 08:43:16.935612 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:43:16.935540    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:43:46.936444 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:43:46.936408    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:44:16.935602 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:44:16.935558    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:44:22.947593 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:22.947561    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpc6gh"]
Nov 03 08:44:22.948017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:22.947852    4473 cpu_manager.go:414] "containerMap: removing container" podUID="1d066ebe-78bd-41e3-b17d-c39567f3efee" containerName="console"
Nov 03 08:44:22.948017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:22.947862    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d066ebe-78bd-41e3-b17d-c39567f3efee" containerName="console"
Nov 03 08:44:22.948017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:22.947925    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d066ebe-78bd-41e3-b17d-c39567f3efee" containerName="console"
Nov 03 08:44:22.950777 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:22.950762    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpc6gh"
Nov 03 08:44:23.053795 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.053758    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkjx7\" (UniqueName: \"kubernetes.io/projected/b9e0325a-4487-46a1-bfe6-1de7a680db9f-kube-api-access-dkjx7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpc6gh\" (UID: \"b9e0325a-4487-46a1-bfe6-1de7a680db9f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpc6gh"
Nov 03 08:44:23.053975 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.053800    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9e0325a-4487-46a1-bfe6-1de7a680db9f-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpc6gh\" (UID: \"b9e0325a-4487-46a1-bfe6-1de7a680db9f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpc6gh"
Nov 03 08:44:23.053975 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.053909    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9e0325a-4487-46a1-bfe6-1de7a680db9f-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpc6gh\" (UID: \"b9e0325a-4487-46a1-bfe6-1de7a680db9f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpc6gh"
Nov 03 08:44:23.054681 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.054662    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 03 08:44:23.054741 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.054663    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 03 08:44:23.054741 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.054686    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-tfctb\""
Nov 03 08:44:23.055717 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.055701    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpc6gh"]
Nov 03 08:44:23.154653 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.154615    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9e0325a-4487-46a1-bfe6-1de7a680db9f-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpc6gh\" (UID: \"b9e0325a-4487-46a1-bfe6-1de7a680db9f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpc6gh"
Nov 03 08:44:23.154827 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.154688    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkjx7\" (UniqueName: \"kubernetes.io/projected/b9e0325a-4487-46a1-bfe6-1de7a680db9f-kube-api-access-dkjx7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpc6gh\" (UID: \"b9e0325a-4487-46a1-bfe6-1de7a680db9f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpc6gh"
Nov 03 08:44:23.154827 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.154727    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9e0325a-4487-46a1-bfe6-1de7a680db9f-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpc6gh\" (UID: \"b9e0325a-4487-46a1-bfe6-1de7a680db9f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpc6gh"
Nov 03 08:44:23.155038 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.155019    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9e0325a-4487-46a1-bfe6-1de7a680db9f-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpc6gh\" (UID: \"b9e0325a-4487-46a1-bfe6-1de7a680db9f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpc6gh"
Nov 03 08:44:23.155126 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.155106    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9e0325a-4487-46a1-bfe6-1de7a680db9f-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpc6gh\" (UID: \"b9e0325a-4487-46a1-bfe6-1de7a680db9f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpc6gh"
Nov 03 08:44:23.215615 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.215553    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkjx7\" (UniqueName: \"kubernetes.io/projected/b9e0325a-4487-46a1-bfe6-1de7a680db9f-kube-api-access-dkjx7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpc6gh\" (UID: \"b9e0325a-4487-46a1-bfe6-1de7a680db9f\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpc6gh"
Nov 03 08:44:23.265388 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.265365    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpc6gh"
Nov 03 08:44:23.451591 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.451560    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv"]
Nov 03 08:44:23.454803 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.454787    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv"
Nov 03 08:44:23.510501 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.510442    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 08:44:23.510501 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.510483    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 08:44:23.510738 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.510721    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 08:44:23.510831 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.510734    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 08:44:23.510831 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.510764    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 08:44:23.510831 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.510774    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 08:44:23.510981 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.510886    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-9gqlj\""
Nov 03 08:44:23.529769 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.526371    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv"]
Nov 03 08:44:23.558116 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.558091    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9db605f6-940f-4654-a666-0d800a3c31fd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-4bwtv\" (UID: \"9db605f6-940f-4654-a666-0d800a3c31fd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv"
Nov 03 08:44:23.558240 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.558122    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9db605f6-940f-4654-a666-0d800a3c31fd-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-4bwtv\" (UID: \"9db605f6-940f-4654-a666-0d800a3c31fd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv"
Nov 03 08:44:23.558240 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.558168    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9db605f6-940f-4654-a666-0d800a3c31fd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-4bwtv\" (UID: \"9db605f6-940f-4654-a666-0d800a3c31fd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv"
Nov 03 08:44:23.558350 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.558329    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbqtq\" (UniqueName: \"kubernetes.io/projected/9db605f6-940f-4654-a666-0d800a3c31fd-kube-api-access-wbqtq\") pod \"istiod-openshift-gateway-7876bfcb74-4bwtv\" (UID: \"9db605f6-940f-4654-a666-0d800a3c31fd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv"
Nov 03 08:44:23.558411 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.558383    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9db605f6-940f-4654-a666-0d800a3c31fd-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-4bwtv\" (UID: \"9db605f6-940f-4654-a666-0d800a3c31fd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv"
Nov 03 08:44:23.558448 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.558432    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db605f6-940f-4654-a666-0d800a3c31fd-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-4bwtv\" (UID: \"9db605f6-940f-4654-a666-0d800a3c31fd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv"
Nov 03 08:44:23.558482 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.558451    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9db605f6-940f-4654-a666-0d800a3c31fd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-4bwtv\" (UID: \"9db605f6-940f-4654-a666-0d800a3c31fd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv"
Nov 03 08:44:23.636145 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.636120    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpc6gh"]
Nov 03 08:44:23.638820 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:44:23.638795    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9e0325a_4487_46a1_bfe6_1de7a680db9f.slice/crio-a17f1ba7d3848a7a6360c0c3d99d16bc8f5b55b478044feb0dc3fc46a9fda6dc WatchSource:0}: Error finding container a17f1ba7d3848a7a6360c0c3d99d16bc8f5b55b478044feb0dc3fc46a9fda6dc: Status 404 returned error can't find the container with id a17f1ba7d3848a7a6360c0c3d99d16bc8f5b55b478044feb0dc3fc46a9fda6dc
Nov 03 08:44:23.659723 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.659697    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbqtq\" (UniqueName: \"kubernetes.io/projected/9db605f6-940f-4654-a666-0d800a3c31fd-kube-api-access-wbqtq\") pod \"istiod-openshift-gateway-7876bfcb74-4bwtv\" (UID: \"9db605f6-940f-4654-a666-0d800a3c31fd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv"
Nov 03 08:44:23.659832 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.659742    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9db605f6-940f-4654-a666-0d800a3c31fd-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-4bwtv\" (UID: \"9db605f6-940f-4654-a666-0d800a3c31fd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv"
Nov 03 08:44:23.659832 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.659770    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db605f6-940f-4654-a666-0d800a3c31fd-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-4bwtv\" (UID: \"9db605f6-940f-4654-a666-0d800a3c31fd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv"
Nov 03 08:44:23.659832 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.659795    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9db605f6-940f-4654-a666-0d800a3c31fd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-4bwtv\" (UID: \"9db605f6-940f-4654-a666-0d800a3c31fd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv"
Nov 03 08:44:23.659832 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.659824    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9db605f6-940f-4654-a666-0d800a3c31fd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-4bwtv\" (UID: \"9db605f6-940f-4654-a666-0d800a3c31fd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv"
Nov 03 08:44:23.660059 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.659850    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9db605f6-940f-4654-a666-0d800a3c31fd-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-4bwtv\" (UID: \"9db605f6-940f-4654-a666-0d800a3c31fd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv"
Nov 03 08:44:23.660059 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.659918    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9db605f6-940f-4654-a666-0d800a3c31fd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-4bwtv\" (UID: \"9db605f6-940f-4654-a666-0d800a3c31fd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv"
Nov 03 08:44:23.660645 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.660523    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9db605f6-940f-4654-a666-0d800a3c31fd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-4bwtv\" (UID: \"9db605f6-940f-4654-a666-0d800a3c31fd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv"
Nov 03 08:44:23.662267 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.662242    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db605f6-940f-4654-a666-0d800a3c31fd-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-4bwtv\" (UID: \"9db605f6-940f-4654-a666-0d800a3c31fd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv"
Nov 03 08:44:23.662431 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.662415    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9db605f6-940f-4654-a666-0d800a3c31fd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-4bwtv\" (UID: \"9db605f6-940f-4654-a666-0d800a3c31fd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv"
Nov 03 08:44:23.662484 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.662440    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9db605f6-940f-4654-a666-0d800a3c31fd-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-4bwtv\" (UID: \"9db605f6-940f-4654-a666-0d800a3c31fd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv"
Nov 03 08:44:23.662610 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.662588    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9db605f6-940f-4654-a666-0d800a3c31fd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-4bwtv\" (UID: \"9db605f6-940f-4654-a666-0d800a3c31fd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv"
Nov 03 08:44:23.718896 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.718869    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9db605f6-940f-4654-a666-0d800a3c31fd-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-4bwtv\" (UID: \"9db605f6-940f-4654-a666-0d800a3c31fd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv"
Nov 03 08:44:23.719241 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.719225    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbqtq\" (UniqueName: \"kubernetes.io/projected/9db605f6-940f-4654-a666-0d800a3c31fd-kube-api-access-wbqtq\") pod \"istiod-openshift-gateway-7876bfcb74-4bwtv\" (UID: \"9db605f6-940f-4654-a666-0d800a3c31fd\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv"
Nov 03 08:44:23.766286 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:23.766244    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv"
Nov 03 08:44:24.003623 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:24.003587    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpc6gh" event={"ID":"b9e0325a-4487-46a1-bfe6-1de7a680db9f","Type":"ContainerStarted","Data":"a17f1ba7d3848a7a6360c0c3d99d16bc8f5b55b478044feb0dc3fc46a9fda6dc"}
Nov 03 08:44:24.190140 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:24.190072    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv"]
Nov 03 08:44:24.192569 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:44:24.192532    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db605f6_940f_4654_a666_0d800a3c31fd.slice/crio-a8209a19ff98a6f27a91a9b0802fbcae35fa8d561c11a34006469313a9ce8981 WatchSource:0}: Error finding container a8209a19ff98a6f27a91a9b0802fbcae35fa8d561c11a34006469313a9ce8981: Status 404 returned error can't find the container with id a8209a19ff98a6f27a91a9b0802fbcae35fa8d561c11a34006469313a9ce8981
Nov 03 08:44:25.009082 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:25.009043    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv" event={"ID":"9db605f6-940f-4654-a666-0d800a3c31fd","Type":"ContainerStarted","Data":"a8209a19ff98a6f27a91a9b0802fbcae35fa8d561c11a34006469313a9ce8981"}
Nov 03 08:44:28.678125 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:28.678091    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 08:44:28.678522 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:28.678169    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 08:44:29.032883 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:29.032279    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv" event={"ID":"9db605f6-940f-4654-a666-0d800a3c31fd","Type":"ContainerStarted","Data":"4a1af2706ab0fccfa373d52a77c139c6fafb67c5b635e85e0fb99cd24c6b0e54"}
Nov 03 08:44:29.032883 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:29.032509    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv"
Nov 03 08:44:29.034652 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:29.034622    4473 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-4bwtv container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 08:44:29.034775 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:29.034694    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv" podUID="9db605f6-940f-4654-a666-0d800a3c31fd" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 08:44:29.243839 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:29.243783    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv" podStartSLOduration=1.7610660839999999 podStartE2EDuration="6.24376377s" totalImagesPullingTime="4.482697686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:44:29.242672092 +0000 UTC m=+293.511735546" watchObservedRunningTime="2026-11-03 08:44:29.24376377 +0000 UTC m=+293.512827204"
Nov 03 08:44:30.038488 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:30.038456    4473 generic.go:356] "Generic (PLEG): container finished" podID="b9e0325a-4487-46a1-bfe6-1de7a680db9f" containerID="4b1bea0dc6ebcc4712af74542b82ad43281aa288b338547550342c297d63eaaf" exitCode=0
Nov 03 08:44:30.038968 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:30.038545    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpc6gh" event={"ID":"b9e0325a-4487-46a1-bfe6-1de7a680db9f","Type":"ContainerDied","Data":"4b1bea0dc6ebcc4712af74542b82ad43281aa288b338547550342c297d63eaaf"}
Nov 03 08:44:30.039458 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:30.039435    4473 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-4bwtv container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 08:44:30.039529 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:30.039478    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv" podUID="9db605f6-940f-4654-a666-0d800a3c31fd" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 08:44:33.040296 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:33.040271    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv"
Nov 03 08:44:34.056472 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:34.056428    4473 generic.go:356] "Generic (PLEG): container finished" podID="b9e0325a-4487-46a1-bfe6-1de7a680db9f" containerID="f77439c49f3c1a41ed081d7b1b894ff83c1ae93513b1df8b3da8f94cb8777b8f" exitCode=0
Nov 03 08:44:34.056879 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:34.056511    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpc6gh" event={"ID":"b9e0325a-4487-46a1-bfe6-1de7a680db9f","Type":"ContainerDied","Data":"f77439c49f3c1a41ed081d7b1b894ff83c1ae93513b1df8b3da8f94cb8777b8f"}
Nov 03 08:44:35.887875 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:35.887850    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 08:44:35.888333 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:35.887917    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 08:44:41.085874 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:41.085841    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpc6gh" event={"ID":"b9e0325a-4487-46a1-bfe6-1de7a680db9f","Type":"ContainerStarted","Data":"75c6966cb7910cfb65190a2c14ab90a524a13048b51cb59622daa7fe55fb3fd9"}
Nov 03 08:44:41.206045 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:41.205994    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpc6gh" podStartSLOduration=3.755679083 podStartE2EDuration="19.205979843s" totalImagesPullingTime="15.45030076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:44:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:44:41.204765872 +0000 UTC m=+305.473829328" watchObservedRunningTime="2026-11-03 08:44:41.205979843 +0000 UTC m=+305.475043276"
Nov 03 08:44:42.091718 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:42.091684    4473 generic.go:356] "Generic (PLEG): container finished" podID="b9e0325a-4487-46a1-bfe6-1de7a680db9f" containerID="75c6966cb7910cfb65190a2c14ab90a524a13048b51cb59622daa7fe55fb3fd9" exitCode=0
Nov 03 08:44:42.092125 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:42.091742    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpc6gh" event={"ID":"b9e0325a-4487-46a1-bfe6-1de7a680db9f","Type":"ContainerDied","Data":"75c6966cb7910cfb65190a2c14ab90a524a13048b51cb59622daa7fe55fb3fd9"}
Nov 03 08:44:43.217718 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:43.217695    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpc6gh"
Nov 03 08:44:43.333810 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:43.333785    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9e0325a-4487-46a1-bfe6-1de7a680db9f-kube-api-access-dkjx7\" (UniqueName: \"kubernetes.io/projected/b9e0325a-4487-46a1-bfe6-1de7a680db9f-kube-api-access-dkjx7\") pod \"b9e0325a-4487-46a1-bfe6-1de7a680db9f\" (UID: \"b9e0325a-4487-46a1-bfe6-1de7a680db9f\") "
Nov 03 08:44:43.333984 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:43.333823    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9e0325a-4487-46a1-bfe6-1de7a680db9f-util\" (UniqueName: \"kubernetes.io/empty-dir/b9e0325a-4487-46a1-bfe6-1de7a680db9f-util\") pod \"b9e0325a-4487-46a1-bfe6-1de7a680db9f\" (UID: \"b9e0325a-4487-46a1-bfe6-1de7a680db9f\") "
Nov 03 08:44:43.333984 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:43.333870    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9e0325a-4487-46a1-bfe6-1de7a680db9f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9e0325a-4487-46a1-bfe6-1de7a680db9f-bundle\") pod \"b9e0325a-4487-46a1-bfe6-1de7a680db9f\" (UID: \"b9e0325a-4487-46a1-bfe6-1de7a680db9f\") "
Nov 03 08:44:43.334503 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:43.334476    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e0325a-4487-46a1-bfe6-1de7a680db9f-bundle" pod "b9e0325a-4487-46a1-bfe6-1de7a680db9f" (UID: "b9e0325a-4487-46a1-bfe6-1de7a680db9f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:44:43.336555 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:43.336523    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9e0325a-4487-46a1-bfe6-1de7a680db9f-kube-api-access-dkjx7" pod "b9e0325a-4487-46a1-bfe6-1de7a680db9f" (UID: "b9e0325a-4487-46a1-bfe6-1de7a680db9f"). InnerVolumeSpecName "kube-api-access-dkjx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:44:43.340281 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:43.340250    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e0325a-4487-46a1-bfe6-1de7a680db9f-util" pod "b9e0325a-4487-46a1-bfe6-1de7a680db9f" (UID: "b9e0325a-4487-46a1-bfe6-1de7a680db9f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:44:43.434719 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:43.434695    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkjx7\" (UniqueName: \"kubernetes.io/projected/b9e0325a-4487-46a1-bfe6-1de7a680db9f-kube-api-access-dkjx7\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:44:43.434719 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:43.434718    4473 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b9e0325a-4487-46a1-bfe6-1de7a680db9f-util\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:44:43.434875 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:43.434728    4473 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b9e0325a-4487-46a1-bfe6-1de7a680db9f-bundle\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:44:44.100995 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:44.100965    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpc6gh" event={"ID":"b9e0325a-4487-46a1-bfe6-1de7a680db9f","Type":"ContainerDied","Data":"a17f1ba7d3848a7a6360c0c3d99d16bc8f5b55b478044feb0dc3fc46a9fda6dc"}
Nov 03 08:44:44.101170 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:44.101014    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a17f1ba7d3848a7a6360c0c3d99d16bc8f5b55b478044feb0dc3fc46a9fda6dc"
Nov 03 08:44:44.101170 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:44:44.100973    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cpc6gh"
Nov 03 08:44:46.936516 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:44:46.936444    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:45:16.935762 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:45:16.935723    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:45:24.924614 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:24.924580    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-r9mf2"]
Nov 03 08:45:24.925141 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:24.924921    4473 cpu_manager.go:414] "containerMap: removing container" podUID="b9e0325a-4487-46a1-bfe6-1de7a680db9f" containerName="util"
Nov 03 08:45:24.925141 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:24.924932    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9e0325a-4487-46a1-bfe6-1de7a680db9f" containerName="util"
Nov 03 08:45:24.925141 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:24.924959    4473 cpu_manager.go:414] "containerMap: removing container" podUID="b9e0325a-4487-46a1-bfe6-1de7a680db9f" containerName="extract"
Nov 03 08:45:24.925141 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:24.924967    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9e0325a-4487-46a1-bfe6-1de7a680db9f" containerName="extract"
Nov 03 08:45:24.925141 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:24.924976    4473 cpu_manager.go:414] "containerMap: removing container" podUID="b9e0325a-4487-46a1-bfe6-1de7a680db9f" containerName="pull"
Nov 03 08:45:24.925141 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:24.924982    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9e0325a-4487-46a1-bfe6-1de7a680db9f" containerName="pull"
Nov 03 08:45:24.925141 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:24.925029    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9e0325a-4487-46a1-bfe6-1de7a680db9f" containerName="extract"
Nov 03 08:45:24.926758 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:24.926741    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-r9mf2"
Nov 03 08:45:24.948477 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:24.948454    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1aede497-0548-4f08-9e87-f24e89e735fe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77df88dcb4-r9mf2\" (UID: \"1aede497-0548-4f08-9e87-f24e89e735fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-r9mf2"
Nov 03 08:45:24.948605 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:24.948485    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1aede497-0548-4f08-9e87-f24e89e735fe-tmp\") pod \"opendatahub-operator-controller-manager-77df88dcb4-r9mf2\" (UID: \"1aede497-0548-4f08-9e87-f24e89e735fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-r9mf2"
Nov 03 08:45:24.948605 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:24.948506    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1aede497-0548-4f08-9e87-f24e89e735fe-webhook-cert\") pod \"opendatahub-operator-controller-manager-77df88dcb4-r9mf2\" (UID: \"1aede497-0548-4f08-9e87-f24e89e735fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-r9mf2"
Nov 03 08:45:24.948797 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:24.948644    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1aede497-0548-4f08-9e87-f24e89e735fe-metrics-cert\") pod \"opendatahub-operator-controller-manager-77df88dcb4-r9mf2\" (UID: \"1aede497-0548-4f08-9e87-f24e89e735fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-r9mf2"
Nov 03 08:45:24.948797 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:24.948683    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njlrx\" (UniqueName: \"kubernetes.io/projected/1aede497-0548-4f08-9e87-f24e89e735fe-kube-api-access-njlrx\") pod \"opendatahub-operator-controller-manager-77df88dcb4-r9mf2\" (UID: \"1aede497-0548-4f08-9e87-f24e89e735fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-r9mf2"
Nov 03 08:45:24.948797 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:24.948702    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1aede497-0548-4f08-9e87-f24e89e735fe-manifests\") pod \"opendatahub-operator-controller-manager-77df88dcb4-r9mf2\" (UID: \"1aede497-0548-4f08-9e87-f24e89e735fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-r9mf2"
Nov 03 08:45:24.987155 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:24.987136    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 08:45:24.987377 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:24.987354    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 08:45:24.987448 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:24.987373    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 08:45:24.987448 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:24.987388    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 08:45:24.987448 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:24.987412    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 08:45:24.987595 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:24.987477    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-tbsxf\""
Nov 03 08:45:25.049792 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:25.049765    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1aede497-0548-4f08-9e87-f24e89e735fe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77df88dcb4-r9mf2\" (UID: \"1aede497-0548-4f08-9e87-f24e89e735fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-r9mf2"
Nov 03 08:45:25.049904 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:25.049794    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1aede497-0548-4f08-9e87-f24e89e735fe-tmp\") pod \"opendatahub-operator-controller-manager-77df88dcb4-r9mf2\" (UID: \"1aede497-0548-4f08-9e87-f24e89e735fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-r9mf2"
Nov 03 08:45:25.049904 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:25.049815    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1aede497-0548-4f08-9e87-f24e89e735fe-webhook-cert\") pod \"opendatahub-operator-controller-manager-77df88dcb4-r9mf2\" (UID: \"1aede497-0548-4f08-9e87-f24e89e735fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-r9mf2"
Nov 03 08:45:25.049904 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:25.049852    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1aede497-0548-4f08-9e87-f24e89e735fe-metrics-cert\") pod \"opendatahub-operator-controller-manager-77df88dcb4-r9mf2\" (UID: \"1aede497-0548-4f08-9e87-f24e89e735fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-r9mf2"
Nov 03 08:45:25.049904 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:25.049890    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njlrx\" (UniqueName: \"kubernetes.io/projected/1aede497-0548-4f08-9e87-f24e89e735fe-kube-api-access-njlrx\") pod \"opendatahub-operator-controller-manager-77df88dcb4-r9mf2\" (UID: \"1aede497-0548-4f08-9e87-f24e89e735fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-r9mf2"
Nov 03 08:45:25.050088 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:25.049916    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1aede497-0548-4f08-9e87-f24e89e735fe-manifests\") pod \"opendatahub-operator-controller-manager-77df88dcb4-r9mf2\" (UID: \"1aede497-0548-4f08-9e87-f24e89e735fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-r9mf2"
Nov 03 08:45:25.050338 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:25.050310    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1aede497-0548-4f08-9e87-f24e89e735fe-manifests\") pod \"opendatahub-operator-controller-manager-77df88dcb4-r9mf2\" (UID: \"1aede497-0548-4f08-9e87-f24e89e735fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-r9mf2"
Nov 03 08:45:25.052438 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:25.052415    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1aede497-0548-4f08-9e87-f24e89e735fe-webhook-cert\") pod \"opendatahub-operator-controller-manager-77df88dcb4-r9mf2\" (UID: \"1aede497-0548-4f08-9e87-f24e89e735fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-r9mf2"
Nov 03 08:45:25.052527 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:25.052483    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1aede497-0548-4f08-9e87-f24e89e735fe-metrics-cert\") pod \"opendatahub-operator-controller-manager-77df88dcb4-r9mf2\" (UID: \"1aede497-0548-4f08-9e87-f24e89e735fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-r9mf2"
Nov 03 08:45:25.052745 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:25.052717    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1aede497-0548-4f08-9e87-f24e89e735fe-tmp\") pod \"opendatahub-operator-controller-manager-77df88dcb4-r9mf2\" (UID: \"1aede497-0548-4f08-9e87-f24e89e735fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-r9mf2"
Nov 03 08:45:25.052847 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:25.052833    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1aede497-0548-4f08-9e87-f24e89e735fe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77df88dcb4-r9mf2\" (UID: \"1aede497-0548-4f08-9e87-f24e89e735fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-r9mf2"
Nov 03 08:45:25.081108 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:25.081084    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-r9mf2"]
Nov 03 08:45:25.119234 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:25.119212    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njlrx\" (UniqueName: \"kubernetes.io/projected/1aede497-0548-4f08-9e87-f24e89e735fe-kube-api-access-njlrx\") pod \"opendatahub-operator-controller-manager-77df88dcb4-r9mf2\" (UID: \"1aede497-0548-4f08-9e87-f24e89e735fe\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-r9mf2"
Nov 03 08:45:25.239373 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:25.239301    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-r9mf2"
Nov 03 08:45:25.726963 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:25.726920    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-r9mf2"]
Nov 03 08:45:25.748673 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:45:25.748643    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aede497_0548_4f08_9e87_f24e89e735fe.slice/crio-5e1d79d43ebb5f5010d09ca53fca3dbf27f13bf84f49306d0e5133626e7f4b75 WatchSource:0}: Error finding container 5e1d79d43ebb5f5010d09ca53fca3dbf27f13bf84f49306d0e5133626e7f4b75: Status 404 returned error can't find the container with id 5e1d79d43ebb5f5010d09ca53fca3dbf27f13bf84f49306d0e5133626e7f4b75
Nov 03 08:45:25.750331 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:25.750312    4473 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 08:45:26.256138 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:26.256101    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-r9mf2" event={"ID":"1aede497-0548-4f08-9e87-f24e89e735fe","Type":"ContainerStarted","Data":"5e1d79d43ebb5f5010d09ca53fca3dbf27f13bf84f49306d0e5133626e7f4b75"}
Nov 03 08:45:29.271277 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:29.271237    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-r9mf2" event={"ID":"1aede497-0548-4f08-9e87-f24e89e735fe","Type":"ContainerStarted","Data":"db2da83d1207dafb89636152a2e4aec580a76ec4ae1b909747970914130caf34"}
Nov 03 08:45:30.276733 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:30.276704    4473 generic.go:356] "Generic (PLEG): container finished" podID="1aede497-0548-4f08-9e87-f24e89e735fe" containerID="db2da83d1207dafb89636152a2e4aec580a76ec4ae1b909747970914130caf34" exitCode=0
Nov 03 08:45:30.277155 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:30.276759    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-r9mf2" event={"ID":"1aede497-0548-4f08-9e87-f24e89e735fe","Type":"ContainerDied","Data":"db2da83d1207dafb89636152a2e4aec580a76ec4ae1b909747970914130caf34"}
Nov 03 08:45:31.282930 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:31.282891    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-r9mf2" event={"ID":"1aede497-0548-4f08-9e87-f24e89e735fe","Type":"ContainerStarted","Data":"e131d2c57644e3b6b4132aa3666983d984d27f4ac10fa7631044f7ff281c3c57"}
Nov 03 08:45:31.283393 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:31.283013    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-r9mf2"
Nov 03 08:45:31.429369 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:31.429320    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-r9mf2" podStartSLOduration=2.819025365 podStartE2EDuration="7.429307581s" totalImagesPullingTime="4.114969942s" totalInitContainerRuntime="495.312274ms" isStatefulPod=false podCreationTimestamp="2026-11-03 08:45:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:45:31.426495498 +0000 UTC m=+355.695558951" watchObservedRunningTime="2026-11-03 08:45:31.429307581 +0000 UTC m=+355.698371014"
Nov 03 08:45:42.290127 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:42.290093    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77df88dcb4-r9mf2"
Nov 03 08:45:46.938051 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:45:46.938018    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:45:52.744395 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:52.744365    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-8p78z"]
Nov 03 08:45:52.747484 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:52.747469    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8p78z"
Nov 03 08:45:52.801116 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:52.801088    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-dr2c8\""
Nov 03 08:45:52.851597 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:52.851573    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 03 08:45:52.851729 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:52.851610    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 03 08:45:52.852883 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:52.852867    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-8p78z"]
Nov 03 08:45:52.881980 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:52.881935    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hg8p\" (UniqueName: \"kubernetes.io/projected/6882d8dd-2f29-4648-bc54-42f039a393e5-kube-api-access-7hg8p\") pod \"cert-manager-webhook-5dbbbf75cc-8p78z\" (UID: \"6882d8dd-2f29-4648-bc54-42f039a393e5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8p78z"
Nov 03 08:45:52.882110 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:52.882093    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6882d8dd-2f29-4648-bc54-42f039a393e5-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-8p78z\" (UID: \"6882d8dd-2f29-4648-bc54-42f039a393e5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8p78z"
Nov 03 08:45:52.983438 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:52.983415    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6882d8dd-2f29-4648-bc54-42f039a393e5-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-8p78z\" (UID: \"6882d8dd-2f29-4648-bc54-42f039a393e5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8p78z"
Nov 03 08:45:52.983558 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:52.983445    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hg8p\" (UniqueName: \"kubernetes.io/projected/6882d8dd-2f29-4648-bc54-42f039a393e5-kube-api-access-7hg8p\") pod \"cert-manager-webhook-5dbbbf75cc-8p78z\" (UID: \"6882d8dd-2f29-4648-bc54-42f039a393e5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8p78z"
Nov 03 08:45:53.052804 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:53.052732    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6882d8dd-2f29-4648-bc54-42f039a393e5-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-8p78z\" (UID: \"6882d8dd-2f29-4648-bc54-42f039a393e5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8p78z"
Nov 03 08:45:53.052898 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:53.052884    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hg8p\" (UniqueName: \"kubernetes.io/projected/6882d8dd-2f29-4648-bc54-42f039a393e5-kube-api-access-7hg8p\") pod \"cert-manager-webhook-5dbbbf75cc-8p78z\" (UID: \"6882d8dd-2f29-4648-bc54-42f039a393e5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8p78z"
Nov 03 08:45:53.069675 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:53.069653    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8p78z"
Nov 03 08:45:53.440189 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:53.440166    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-8p78z"]
Nov 03 08:45:53.442718 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:45:53.442691    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6882d8dd_2f29_4648_bc54_42f039a393e5.slice/crio-05a18a4b3a3088fcf1d31bd43b4211c9fdfc0c0f5de5e28839e06507c5a5d62a WatchSource:0}: Error finding container 05a18a4b3a3088fcf1d31bd43b4211c9fdfc0c0f5de5e28839e06507c5a5d62a: Status 404 returned error can't find the container with id 05a18a4b3a3088fcf1d31bd43b4211c9fdfc0c0f5de5e28839e06507c5a5d62a
Nov 03 08:45:54.380205 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:54.380174    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8p78z" event={"ID":"6882d8dd-2f29-4648-bc54-42f039a393e5","Type":"ContainerStarted","Data":"05a18a4b3a3088fcf1d31bd43b4211c9fdfc0c0f5de5e28839e06507c5a5d62a"}
Nov 03 08:45:56.856116 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:56.856080    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-677799c6bb-fftnv"]
Nov 03 08:45:56.860474 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:56.860453    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-677799c6bb-fftnv"
Nov 03 08:45:56.918223 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:56.918185    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-w8mcg\""
Nov 03 08:45:56.920370 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:56.920277    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-677799c6bb-fftnv"]
Nov 03 08:45:57.011244 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:57.011192    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 03 08:45:57.011244 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:57.011245    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 03 08:45:57.017345 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:57.017321    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e9796dc-2aa9-4c4b-a171-dee0cb98ca23-tls-certs\") pod \"kube-auth-proxy-677799c6bb-fftnv\" (UID: \"2e9796dc-2aa9-4c4b-a171-dee0cb98ca23\") " pod="openshift-ingress/kube-auth-proxy-677799c6bb-fftnv"
Nov 03 08:45:57.017474 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:57.017372    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qszkf\" (UniqueName: \"kubernetes.io/projected/2e9796dc-2aa9-4c4b-a171-dee0cb98ca23-kube-api-access-qszkf\") pod \"kube-auth-proxy-677799c6bb-fftnv\" (UID: \"2e9796dc-2aa9-4c4b-a171-dee0cb98ca23\") " pod="openshift-ingress/kube-auth-proxy-677799c6bb-fftnv"
Nov 03 08:45:57.017474 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:57.017447    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e9796dc-2aa9-4c4b-a171-dee0cb98ca23-tmp\") pod \"kube-auth-proxy-677799c6bb-fftnv\" (UID: \"2e9796dc-2aa9-4c4b-a171-dee0cb98ca23\") " pod="openshift-ingress/kube-auth-proxy-677799c6bb-fftnv"
Nov 03 08:45:57.117923 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:57.117902    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e9796dc-2aa9-4c4b-a171-dee0cb98ca23-tmp\") pod \"kube-auth-proxy-677799c6bb-fftnv\" (UID: \"2e9796dc-2aa9-4c4b-a171-dee0cb98ca23\") " pod="openshift-ingress/kube-auth-proxy-677799c6bb-fftnv"
Nov 03 08:45:57.118045 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:57.117972    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e9796dc-2aa9-4c4b-a171-dee0cb98ca23-tls-certs\") pod \"kube-auth-proxy-677799c6bb-fftnv\" (UID: \"2e9796dc-2aa9-4c4b-a171-dee0cb98ca23\") " pod="openshift-ingress/kube-auth-proxy-677799c6bb-fftnv"
Nov 03 08:45:57.118090 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:57.118047    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qszkf\" (UniqueName: \"kubernetes.io/projected/2e9796dc-2aa9-4c4b-a171-dee0cb98ca23-kube-api-access-qszkf\") pod \"kube-auth-proxy-677799c6bb-fftnv\" (UID: \"2e9796dc-2aa9-4c4b-a171-dee0cb98ca23\") " pod="openshift-ingress/kube-auth-proxy-677799c6bb-fftnv"
Nov 03 08:45:57.120317 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:57.120293    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e9796dc-2aa9-4c4b-a171-dee0cb98ca23-tmp\") pod \"kube-auth-proxy-677799c6bb-fftnv\" (UID: \"2e9796dc-2aa9-4c4b-a171-dee0cb98ca23\") " pod="openshift-ingress/kube-auth-proxy-677799c6bb-fftnv"
Nov 03 08:45:57.120439 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:57.120422    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e9796dc-2aa9-4c4b-a171-dee0cb98ca23-tls-certs\") pod \"kube-auth-proxy-677799c6bb-fftnv\" (UID: \"2e9796dc-2aa9-4c4b-a171-dee0cb98ca23\") " pod="openshift-ingress/kube-auth-proxy-677799c6bb-fftnv"
Nov 03 08:45:57.180479 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:57.180447    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qszkf\" (UniqueName: \"kubernetes.io/projected/2e9796dc-2aa9-4c4b-a171-dee0cb98ca23-kube-api-access-qszkf\") pod \"kube-auth-proxy-677799c6bb-fftnv\" (UID: \"2e9796dc-2aa9-4c4b-a171-dee0cb98ca23\") " pod="openshift-ingress/kube-auth-proxy-677799c6bb-fftnv"
Nov 03 08:45:57.395845 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:57.395752    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8p78z" event={"ID":"6882d8dd-2f29-4648-bc54-42f039a393e5","Type":"ContainerStarted","Data":"32782261126f4d7a5baccacdb9fc41ba572782371e446971c8b8f45aa4d971e9"}
Nov 03 08:45:57.396030 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:57.395877    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8p78z"
Nov 03 08:45:57.474545 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:57.474516    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-677799c6bb-fftnv"
Nov 03 08:45:57.859083 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:57.858996    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8p78z" podStartSLOduration=2.223671125 podStartE2EDuration="5.858979454s" totalImagesPullingTime="3.635308329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:45:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:45:57.519142131 +0000 UTC m=+381.788205582" watchObservedRunningTime="2026-11-03 08:45:57.858979454 +0000 UTC m=+382.128042888"
Nov 03 08:45:57.860135 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:57.860118    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-677799c6bb-fftnv"]
Nov 03 08:45:57.862023 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:45:57.861999    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e9796dc_2aa9_4c4b_a171_dee0cb98ca23.slice/crio-db8c15964a84a0c1353fad98c81714d523372b11edaf61ad3b6c878b8d239492 WatchSource:0}: Error finding container db8c15964a84a0c1353fad98c81714d523372b11edaf61ad3b6c878b8d239492: Status 404 returned error can't find the container with id db8c15964a84a0c1353fad98c81714d523372b11edaf61ad3b6c878b8d239492
Nov 03 08:45:58.401032 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:45:58.401001    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-677799c6bb-fftnv" event={"ID":"2e9796dc-2aa9-4c4b-a171-dee0cb98ca23","Type":"ContainerStarted","Data":"db8c15964a84a0c1353fad98c81714d523372b11edaf61ad3b6c878b8d239492"}
Nov 03 08:46:00.801352 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:00.801322    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 08:46:00.804619 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:00.804601    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"
Nov 03 08:46:00.860307 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:00.860229    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-znzl4\""
Nov 03 08:46:00.862511 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:00.862479    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 08:46:00.910697 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:00.910667    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 08:46:00.910697 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:00.910692    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 03 08:46:00.910882 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:00.910692    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 08:46:00.952813 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:00.952787    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2cgw\" (UniqueName: \"kubernetes.io/projected/a025cb89-c80d-48f7-9f0a-93c8b6c24950-kube-api-access-l2cgw\") pod \"limitador-limitador-7c9b87d574-xbn6s\" (UID: \"a025cb89-c80d-48f7-9f0a-93c8b6c24950\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"
Nov 03 08:46:00.952996 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:00.952845    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a025cb89-c80d-48f7-9f0a-93c8b6c24950-config-file\") pod \"limitador-limitador-7c9b87d574-xbn6s\" (UID: \"a025cb89-c80d-48f7-9f0a-93c8b6c24950\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"
Nov 03 08:46:01.054079 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:01.054047    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a025cb89-c80d-48f7-9f0a-93c8b6c24950-config-file\") pod \"limitador-limitador-7c9b87d574-xbn6s\" (UID: \"a025cb89-c80d-48f7-9f0a-93c8b6c24950\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"
Nov 03 08:46:01.054244 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:01.054102    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2cgw\" (UniqueName: \"kubernetes.io/projected/a025cb89-c80d-48f7-9f0a-93c8b6c24950-kube-api-access-l2cgw\") pod \"limitador-limitador-7c9b87d574-xbn6s\" (UID: \"a025cb89-c80d-48f7-9f0a-93c8b6c24950\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"
Nov 03 08:46:01.054770 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:01.054746    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a025cb89-c80d-48f7-9f0a-93c8b6c24950-config-file\") pod \"limitador-limitador-7c9b87d574-xbn6s\" (UID: \"a025cb89-c80d-48f7-9f0a-93c8b6c24950\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"
Nov 03 08:46:01.107247 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:01.107219    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 08:46:01.113609 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:01.113547    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2cgw\" (UniqueName: \"kubernetes.io/projected/a025cb89-c80d-48f7-9f0a-93c8b6c24950-kube-api-access-l2cgw\") pod \"limitador-limitador-7c9b87d574-xbn6s\" (UID: \"a025cb89-c80d-48f7-9f0a-93c8b6c24950\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"
Nov 03 08:46:01.121398 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:01.121378    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"
Nov 03 08:46:01.416918 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:01.416883    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-677799c6bb-fftnv" event={"ID":"2e9796dc-2aa9-4c4b-a171-dee0cb98ca23","Type":"ContainerStarted","Data":"1085749d0ea68a93bc4a3c896fd5f59bcf812188b6b714847fdfefb111ddeed4"}
Nov 03 08:46:01.492931 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:01.492826    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 08:46:01.495538 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:46:01.495511    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda025cb89_c80d_48f7_9f0a_93c8b6c24950.slice/crio-85b566ed760468c1fb68cd3bc9510c8597c5d8f33a56099913084a780ac627c5 WatchSource:0}: Error finding container 85b566ed760468c1fb68cd3bc9510c8597c5d8f33a56099913084a780ac627c5: Status 404 returned error can't find the container with id 85b566ed760468c1fb68cd3bc9510c8597c5d8f33a56099913084a780ac627c5
Nov 03 08:46:01.533587 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:01.533539    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-677799c6bb-fftnv" podStartSLOduration=2.8268469229999997 podStartE2EDuration="5.533524933s" totalImagesPullingTime="2.70667801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:45:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:46:01.53222536 +0000 UTC m=+385.801288815" watchObservedRunningTime="2026-11-03 08:46:01.533524933 +0000 UTC m=+385.802588411"
Nov 03 08:46:02.422165 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:02.422131    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-xbn6s" event={"ID":"a025cb89-c80d-48f7-9f0a-93c8b6c24950","Type":"ContainerStarted","Data":"85b566ed760468c1fb68cd3bc9510c8597c5d8f33a56099913084a780ac627c5"}
Nov 03 08:46:03.404385 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:03.404356    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8p78z"
Nov 03 08:46:05.436398 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:05.436361    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-xbn6s" event={"ID":"a025cb89-c80d-48f7-9f0a-93c8b6c24950","Type":"ContainerStarted","Data":"5584cffc575e66aec6a31ec9fe37851d15d3b1072cfde22025cceb90343fc98e"}
Nov 03 08:46:05.436789 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:05.436441    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"
Nov 03 08:46:05.554194 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:05.554136    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-xbn6s" podStartSLOduration=2.589393971 podStartE2EDuration="5.554121306s" totalImagesPullingTime="2.964727335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:46:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:46:05.552670504 +0000 UTC m=+389.821733995" watchObservedRunningTime="2026-11-03 08:46:05.554121306 +0000 UTC m=+389.823184781"
Nov 03 08:46:16.441635 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:16.441602    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"
Nov 03 08:46:16.940891 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:46:16.940796    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:46:46.936704 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:46:46.936659    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:46:52.886252 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:52.886218    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7dd5f74f7b-lk7c9"]
Nov 03 08:46:52.888156 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:52.888137    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dd5f74f7b-lk7c9"
Nov 03 08:46:52.964272 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:52.964244    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp5m5\" (UniqueName: \"kubernetes.io/projected/17e69a7f-4e64-4aad-b798-af8fa74a2b76-kube-api-access-pp5m5\") pod \"maas-controller-7dd5f74f7b-lk7c9\" (UID: \"17e69a7f-4e64-4aad-b798-af8fa74a2b76\") " pod="opendatahub/maas-controller-7dd5f74f7b-lk7c9"
Nov 03 08:46:52.964426 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:52.964304    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17e69a7f-4e64-4aad-b798-af8fa74a2b76-metrics-certs\") pod \"maas-controller-7dd5f74f7b-lk7c9\" (UID: \"17e69a7f-4e64-4aad-b798-af8fa74a2b76\") " pod="opendatahub/maas-controller-7dd5f74f7b-lk7c9"
Nov 03 08:46:52.964476 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:52.964460    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/17e69a7f-4e64-4aad-b798-af8fa74a2b76-webhook-cert\") pod \"maas-controller-7dd5f74f7b-lk7c9\" (UID: \"17e69a7f-4e64-4aad-b798-af8fa74a2b76\") " pod="opendatahub/maas-controller-7dd5f74f7b-lk7c9"
Nov 03 08:46:52.992568 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:52.992540    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 08:46:52.992568 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:52.992540    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-k54gs\""
Nov 03 08:46:52.992771 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:52.992548    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 08:46:52.992771 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:52.992581    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 08:46:52.992771 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:52.992613    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 03 08:46:52.992771 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:52.992703    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 03 08:46:52.994759 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:52.994739    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dd5f74f7b-lk7c9"]
Nov 03 08:46:53.065331 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:53.065307    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17e69a7f-4e64-4aad-b798-af8fa74a2b76-metrics-certs\") pod \"maas-controller-7dd5f74f7b-lk7c9\" (UID: \"17e69a7f-4e64-4aad-b798-af8fa74a2b76\") " pod="opendatahub/maas-controller-7dd5f74f7b-lk7c9"
Nov 03 08:46:53.065466 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:53.065355    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/17e69a7f-4e64-4aad-b798-af8fa74a2b76-webhook-cert\") pod \"maas-controller-7dd5f74f7b-lk7c9\" (UID: \"17e69a7f-4e64-4aad-b798-af8fa74a2b76\") " pod="opendatahub/maas-controller-7dd5f74f7b-lk7c9"
Nov 03 08:46:53.065466 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:53.065386    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp5m5\" (UniqueName: \"kubernetes.io/projected/17e69a7f-4e64-4aad-b798-af8fa74a2b76-kube-api-access-pp5m5\") pod \"maas-controller-7dd5f74f7b-lk7c9\" (UID: \"17e69a7f-4e64-4aad-b798-af8fa74a2b76\") " pod="opendatahub/maas-controller-7dd5f74f7b-lk7c9"
Nov 03 08:46:53.067796 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:53.067768    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/17e69a7f-4e64-4aad-b798-af8fa74a2b76-webhook-cert\") pod \"maas-controller-7dd5f74f7b-lk7c9\" (UID: \"17e69a7f-4e64-4aad-b798-af8fa74a2b76\") " pod="opendatahub/maas-controller-7dd5f74f7b-lk7c9"
Nov 03 08:46:53.067923 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:53.067901    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17e69a7f-4e64-4aad-b798-af8fa74a2b76-metrics-certs\") pod \"maas-controller-7dd5f74f7b-lk7c9\" (UID: \"17e69a7f-4e64-4aad-b798-af8fa74a2b76\") " pod="opendatahub/maas-controller-7dd5f74f7b-lk7c9"
Nov 03 08:46:53.124859 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:53.124836    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp5m5\" (UniqueName: \"kubernetes.io/projected/17e69a7f-4e64-4aad-b798-af8fa74a2b76-kube-api-access-pp5m5\") pod \"maas-controller-7dd5f74f7b-lk7c9\" (UID: \"17e69a7f-4e64-4aad-b798-af8fa74a2b76\") " pod="opendatahub/maas-controller-7dd5f74f7b-lk7c9"
Nov 03 08:46:53.199699 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:53.199625    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dd5f74f7b-lk7c9"
Nov 03 08:46:53.569461 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:53.569439    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dd5f74f7b-lk7c9"]
Nov 03 08:46:53.571929 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:46:53.571903    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e69a7f_4e64_4aad_b798_af8fa74a2b76.slice/crio-c26f3e46f41fe04330875db625dd78e966fccc824975db918465880400e702c0 WatchSource:0}: Error finding container c26f3e46f41fe04330875db625dd78e966fccc824975db918465880400e702c0: Status 404 returned error can't find the container with id c26f3e46f41fe04330875db625dd78e966fccc824975db918465880400e702c0
Nov 03 08:46:53.627897 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:53.627870    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dd5f74f7b-lk7c9" event={"ID":"17e69a7f-4e64-4aad-b798-af8fa74a2b76","Type":"ContainerStarted","Data":"c26f3e46f41fe04330875db625dd78e966fccc824975db918465880400e702c0"}
Nov 03 08:46:59.655020 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:59.654983    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dd5f74f7b-lk7c9" event={"ID":"17e69a7f-4e64-4aad-b798-af8fa74a2b76","Type":"ContainerStarted","Data":"367fb63de7da5a3a8d40ca0f481177808f079ee612f30457884b3c459ae49741"}
Nov 03 08:46:59.655395 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:59.655190    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7dd5f74f7b-lk7c9"
Nov 03 08:46:59.805488 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:46:59.805380    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7dd5f74f7b-lk7c9" podStartSLOduration=1.855600404 podStartE2EDuration="7.805362786s" totalImagesPullingTime="5.949762382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:46:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:46:59.804930877 +0000 UTC m=+444.073994331" watchObservedRunningTime="2026-11-03 08:46:59.805362786 +0000 UTC m=+444.074426220"
Nov 03 08:47:02.887216 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:02.887181    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p72nd"]
Nov 03 08:47:02.889218 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:02.889203    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p72nd"
Nov 03 08:47:02.942706 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:02.942675    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Nov 03 08:47:02.942926 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:02.942911    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Nov 03 08:47:02.944280 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:02.944263    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Nov 03 08:47:02.950805 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:02.950775    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sb79\" (UniqueName: \"kubernetes.io/projected/dc80f365-2a40-4fd0-ae2b-6cfc815a3d63-kube-api-access-2sb79\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-p72nd\" (UID: \"dc80f365-2a40-4fd0-ae2b-6cfc815a3d63\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p72nd"
Nov 03 08:47:02.950919 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:02.950824    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc80f365-2a40-4fd0-ae2b-6cfc815a3d63-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-p72nd\" (UID: \"dc80f365-2a40-4fd0-ae2b-6cfc815a3d63\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p72nd"
Nov 03 08:47:02.950995 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:02.950968    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/dc80f365-2a40-4fd0-ae2b-6cfc815a3d63-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-p72nd\" (UID: \"dc80f365-2a40-4fd0-ae2b-6cfc815a3d63\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p72nd"
Nov 03 08:47:02.951060 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:02.951045    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dc80f365-2a40-4fd0-ae2b-6cfc815a3d63-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-p72nd\" (UID: \"dc80f365-2a40-4fd0-ae2b-6cfc815a3d63\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p72nd"
Nov 03 08:47:02.993320 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:02.993276    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-9lmgm\""
Nov 03 08:47:02.994795 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:02.994775    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p72nd"]
Nov 03 08:47:03.051998 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:03.051938    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sb79\" (UniqueName: \"kubernetes.io/projected/dc80f365-2a40-4fd0-ae2b-6cfc815a3d63-kube-api-access-2sb79\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-p72nd\" (UID: \"dc80f365-2a40-4fd0-ae2b-6cfc815a3d63\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p72nd"
Nov 03 08:47:03.052188 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:03.052004    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc80f365-2a40-4fd0-ae2b-6cfc815a3d63-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-p72nd\" (UID: \"dc80f365-2a40-4fd0-ae2b-6cfc815a3d63\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p72nd"
Nov 03 08:47:03.052188 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:03.052033    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/dc80f365-2a40-4fd0-ae2b-6cfc815a3d63-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-p72nd\" (UID: \"dc80f365-2a40-4fd0-ae2b-6cfc815a3d63\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p72nd"
Nov 03 08:47:03.052188 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:03.052076    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dc80f365-2a40-4fd0-ae2b-6cfc815a3d63-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-p72nd\" (UID: \"dc80f365-2a40-4fd0-ae2b-6cfc815a3d63\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p72nd"
Nov 03 08:47:03.052188 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:47:03.052160    4473 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found
Nov 03 08:47:03.052188 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:47:03.052189    4473 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found
Nov 03 08:47:03.052470 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:47:03.052241    4473 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dc80f365-2a40-4fd0-ae2b-6cfc815a3d63-proxy-source podName:dc80f365-2a40-4fd0-ae2b-6cfc815a3d63 nodeName:}" failed. No retries permitted until 2026-11-03 08:47:03.552219828 +0000 UTC m=+447.821283254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/dc80f365-2a40-4fd0-ae2b-6cfc815a3d63-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-p72nd" (UID: "dc80f365-2a40-4fd0-ae2b-6cfc815a3d63") : configmap "usage-tenancy-proxy-source" not found
Nov 03 08:47:03.052470 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:47:03.052260    4473 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc80f365-2a40-4fd0-ae2b-6cfc815a3d63-tls-cert podName:dc80f365-2a40-4fd0-ae2b-6cfc815a3d63 nodeName:}" failed. No retries permitted until 2026-11-03 08:47:03.552250686 +0000 UTC m=+447.821314101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/dc80f365-2a40-4fd0-ae2b-6cfc815a3d63-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-p72nd" (UID: "dc80f365-2a40-4fd0-ae2b-6cfc815a3d63") : secret "usage-logs-tenancy-proxy-tls" not found
Nov 03 08:47:03.052638 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:03.052621    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dc80f365-2a40-4fd0-ae2b-6cfc815a3d63-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-p72nd\" (UID: \"dc80f365-2a40-4fd0-ae2b-6cfc815a3d63\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p72nd"
Nov 03 08:47:03.124455 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:03.124430    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sb79\" (UniqueName: \"kubernetes.io/projected/dc80f365-2a40-4fd0-ae2b-6cfc815a3d63-kube-api-access-2sb79\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-p72nd\" (UID: \"dc80f365-2a40-4fd0-ae2b-6cfc815a3d63\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p72nd"
Nov 03 08:47:03.555845 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:03.555812    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/dc80f365-2a40-4fd0-ae2b-6cfc815a3d63-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-p72nd\" (UID: \"dc80f365-2a40-4fd0-ae2b-6cfc815a3d63\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p72nd"
Nov 03 08:47:03.556048 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:03.555866    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dc80f365-2a40-4fd0-ae2b-6cfc815a3d63-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-p72nd\" (UID: \"dc80f365-2a40-4fd0-ae2b-6cfc815a3d63\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p72nd"
Nov 03 08:47:03.557051 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:03.557014    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/dc80f365-2a40-4fd0-ae2b-6cfc815a3d63-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-p72nd\" (UID: \"dc80f365-2a40-4fd0-ae2b-6cfc815a3d63\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p72nd"
Nov 03 08:47:03.558483 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:03.558460    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dc80f365-2a40-4fd0-ae2b-6cfc815a3d63-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-p72nd\" (UID: \"dc80f365-2a40-4fd0-ae2b-6cfc815a3d63\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p72nd"
Nov 03 08:47:03.800291 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:03.800259    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p72nd"
Nov 03 08:47:04.175761 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:04.175738    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p72nd"]
Nov 03 08:47:04.177845 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:47:04.177820    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc80f365_2a40_4fd0_ae2b_6cfc815a3d63.slice/crio-dd1bd9b6aca7a6d478335e42c53187c3a59ca9dd9c994aa9f0162e68b8857cf0 WatchSource:0}: Error finding container dd1bd9b6aca7a6d478335e42c53187c3a59ca9dd9c994aa9f0162e68b8857cf0: Status 404 returned error can't find the container with id dd1bd9b6aca7a6d478335e42c53187c3a59ca9dd9c994aa9f0162e68b8857cf0
Nov 03 08:47:04.677410 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:04.677377    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p72nd" event={"ID":"dc80f365-2a40-4fd0-ae2b-6cfc815a3d63","Type":"ContainerStarted","Data":"dd1bd9b6aca7a6d478335e42c53187c3a59ca9dd9c994aa9f0162e68b8857cf0"}
Nov 03 08:47:10.661690 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:10.661662    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7dd5f74f7b-lk7c9"
Nov 03 08:47:11.838643 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:11.838609    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x2f6z"]
Nov 03 08:47:11.843958 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:11.843930    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x2f6z"
Nov 03 08:47:11.904886 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:11.904854    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x2f6z"]
Nov 03 08:47:11.931646 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:11.931619    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x2f6z\" (UID: \"269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x2f6z"
Nov 03 08:47:11.931820 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:11.931663    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x2f6z\" (UID: \"269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x2f6z"
Nov 03 08:47:11.931820 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:11.931690    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x2f6z\" (UID: \"269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x2f6z"
Nov 03 08:47:11.931820 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:11.931765    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv5f4\" (UniqueName: \"kubernetes.io/projected/269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6-kube-api-access-jv5f4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x2f6z\" (UID: \"269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x2f6z"
Nov 03 08:47:11.932031 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:11.931862    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x2f6z\" (UID: \"269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x2f6z"
Nov 03 08:47:11.932031 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:11.931935    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x2f6z\" (UID: \"269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x2f6z"
Nov 03 08:47:11.932031 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:11.932006    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x2f6z\" (UID: \"269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x2f6z"
Nov 03 08:47:12.032515 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:12.032486    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x2f6z\" (UID: \"269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x2f6z"
Nov 03 08:47:12.032701 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:12.032542    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x2f6z\" (UID: \"269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x2f6z"
Nov 03 08:47:12.032785 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:12.032732    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x2f6z\" (UID: \"269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x2f6z"
Nov 03 08:47:12.032785 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:12.032774    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv5f4\" (UniqueName: \"kubernetes.io/projected/269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6-kube-api-access-jv5f4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x2f6z\" (UID: \"269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x2f6z"
Nov 03 08:47:12.032870 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:12.032808    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x2f6z\" (UID: \"269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x2f6z"
Nov 03 08:47:12.032923 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:12.032886    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x2f6z\" (UID: \"269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x2f6z"
Nov 03 08:47:12.033011 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:12.032930    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x2f6z\" (UID: \"269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x2f6z"
Nov 03 08:47:12.033667 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:12.033641    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x2f6z\" (UID: \"269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x2f6z"
Nov 03 08:47:12.035279 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:12.035251    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x2f6z\" (UID: \"269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x2f6z"
Nov 03 08:47:12.036208 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:12.036182    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x2f6z\" (UID: \"269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x2f6z"
Nov 03 08:47:12.036327 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:12.036264    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x2f6z\" (UID: \"269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x2f6z"
Nov 03 08:47:12.036391 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:12.036355    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x2f6z\" (UID: \"269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x2f6z"
Nov 03 08:47:12.098246 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:12.098174    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv5f4\" (UniqueName: \"kubernetes.io/projected/269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6-kube-api-access-jv5f4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x2f6z\" (UID: \"269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x2f6z"
Nov 03 08:47:12.098410 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:12.098383    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-x2f6z\" (UID: \"269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x2f6z"
Nov 03 08:47:12.159561 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:12.159535    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x2f6z"
Nov 03 08:47:12.591623 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:12.591591    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x2f6z"]
Nov 03 08:47:12.595516 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:47:12.595489    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod269aa896_4fbc_4cdb_ac6e_1f2978fc6bd6.slice/crio-120bf8b6be9b9288203aedb4d0e277d3fae005bbbb87d7165d7493d54d973739 WatchSource:0}: Error finding container 120bf8b6be9b9288203aedb4d0e277d3fae005bbbb87d7165d7493d54d973739: Status 404 returned error can't find the container with id 120bf8b6be9b9288203aedb4d0e277d3fae005bbbb87d7165d7493d54d973739
Nov 03 08:47:12.598040 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:12.597929    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 08:47:12.598140 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:12.598097    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 08:47:12.722352 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:12.721500    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x2f6z" event={"ID":"269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6","Type":"ContainerStarted","Data":"603899393d231ca544885eafece799b9d199efa548b802e12cf051bc706802d2"}
Nov 03 08:47:12.722352 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:12.721565    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x2f6z" event={"ID":"269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6","Type":"ContainerStarted","Data":"120bf8b6be9b9288203aedb4d0e277d3fae005bbbb87d7165d7493d54d973739"}
Nov 03 08:47:12.722352 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:12.722306    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x2f6z"
Nov 03 08:47:12.843495 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:12.843381    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x2f6z" podStartSLOduration=1.8433634909999999 podStartE2EDuration="1.843363491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:47:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:47:12.842083384 +0000 UTC m=+457.111146820" watchObservedRunningTime="2026-11-03 08:47:12.843363491 +0000 UTC m=+457.112426933"
Nov 03 08:47:13.728214 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:13.728180    4473 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-x2f6z container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 08:47:13.728380 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:13.728234    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x2f6z" podUID="269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 08:47:14.732371 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:14.732342    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-x2f6z"
Nov 03 08:47:14.956928 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:14.956896    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv"]
Nov 03 08:47:14.957207 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:14.957179    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv" podUID="9db605f6-940f-4654-a666-0d800a3c31fd" containerName="discovery" containerID="cri-o://4a1af2706ab0fccfa373d52a77c139c6fafb67c5b635e85e0fb99cd24c6b0e54" gracePeriod=30
Nov 03 08:47:15.039775 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:15.039747    4473 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-4bwtv container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.22:8080/ready\": dial tcp 10.133.0.22:8080: connect: connection refused" start-of-body=
Nov 03 08:47:15.039928 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:15.039818    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv" podUID="9db605f6-940f-4654-a666-0d800a3c31fd" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.22:8080/ready\": dial tcp 10.133.0.22:8080: connect: connection refused"
Nov 03 08:47:15.219858 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:15.219830    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv"
Nov 03 08:47:15.365013 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:15.364896    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9db605f6-940f-4654-a666-0d800a3c31fd-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9db605f6-940f-4654-a666-0d800a3c31fd-istio-csr-dns-cert\") pod \"9db605f6-940f-4654-a666-0d800a3c31fd\" (UID: \"9db605f6-940f-4654-a666-0d800a3c31fd\") "
Nov 03 08:47:15.365013 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:15.364990    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9db605f6-940f-4654-a666-0d800a3c31fd-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9db605f6-940f-4654-a666-0d800a3c31fd-istio-csr-ca-configmap\") pod \"9db605f6-940f-4654-a666-0d800a3c31fd\" (UID: \"9db605f6-940f-4654-a666-0d800a3c31fd\") "
Nov 03 08:47:15.365237 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:15.365061    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db605f6-940f-4654-a666-0d800a3c31fd-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db605f6-940f-4654-a666-0d800a3c31fd-local-certs\") pod \"9db605f6-940f-4654-a666-0d800a3c31fd\" (UID: \"9db605f6-940f-4654-a666-0d800a3c31fd\") "
Nov 03 08:47:15.365237 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:15.365099    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9db605f6-940f-4654-a666-0d800a3c31fd-istio-token\" (UniqueName: \"kubernetes.io/projected/9db605f6-940f-4654-a666-0d800a3c31fd-istio-token\") pod \"9db605f6-940f-4654-a666-0d800a3c31fd\" (UID: \"9db605f6-940f-4654-a666-0d800a3c31fd\") "
Nov 03 08:47:15.365237 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:15.365157    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9db605f6-940f-4654-a666-0d800a3c31fd-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9db605f6-940f-4654-a666-0d800a3c31fd-istio-kubeconfig\") pod \"9db605f6-940f-4654-a666-0d800a3c31fd\" (UID: \"9db605f6-940f-4654-a666-0d800a3c31fd\") "
Nov 03 08:47:15.365237 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:15.365194    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9db605f6-940f-4654-a666-0d800a3c31fd-kube-api-access-wbqtq\" (UniqueName: \"kubernetes.io/projected/9db605f6-940f-4654-a666-0d800a3c31fd-kube-api-access-wbqtq\") pod \"9db605f6-940f-4654-a666-0d800a3c31fd\" (UID: \"9db605f6-940f-4654-a666-0d800a3c31fd\") "
Nov 03 08:47:15.365237 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:15.365222    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9db605f6-940f-4654-a666-0d800a3c31fd-cacerts\" (UniqueName: \"kubernetes.io/secret/9db605f6-940f-4654-a666-0d800a3c31fd-cacerts\") pod \"9db605f6-940f-4654-a666-0d800a3c31fd\" (UID: \"9db605f6-940f-4654-a666-0d800a3c31fd\") "
Nov 03 08:47:15.366228 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:15.365995    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9db605f6-940f-4654-a666-0d800a3c31fd-istio-csr-ca-configmap" pod "9db605f6-940f-4654-a666-0d800a3c31fd" (UID: "9db605f6-940f-4654-a666-0d800a3c31fd"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:47:15.368466 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:15.368431    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9db605f6-940f-4654-a666-0d800a3c31fd-istio-csr-dns-cert" pod "9db605f6-940f-4654-a666-0d800a3c31fd" (UID: "9db605f6-940f-4654-a666-0d800a3c31fd"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:47:15.368612 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:15.368581    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9db605f6-940f-4654-a666-0d800a3c31fd-cacerts" pod "9db605f6-940f-4654-a666-0d800a3c31fd" (UID: "9db605f6-940f-4654-a666-0d800a3c31fd"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:47:15.368883 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:15.368847    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db605f6-940f-4654-a666-0d800a3c31fd-istio-token" pod "9db605f6-940f-4654-a666-0d800a3c31fd" (UID: "9db605f6-940f-4654-a666-0d800a3c31fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:47:15.369044 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:15.368969    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db605f6-940f-4654-a666-0d800a3c31fd-local-certs" pod "9db605f6-940f-4654-a666-0d800a3c31fd" (UID: "9db605f6-940f-4654-a666-0d800a3c31fd"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:47:15.370076 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:15.370027    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9db605f6-940f-4654-a666-0d800a3c31fd-istio-kubeconfig" pod "9db605f6-940f-4654-a666-0d800a3c31fd" (UID: "9db605f6-940f-4654-a666-0d800a3c31fd"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:47:15.375436 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:15.375403    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db605f6-940f-4654-a666-0d800a3c31fd-kube-api-access-wbqtq" pod "9db605f6-940f-4654-a666-0d800a3c31fd" (UID: "9db605f6-940f-4654-a666-0d800a3c31fd"). InnerVolumeSpecName "kube-api-access-wbqtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:47:15.466338 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:15.466298    4473 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db605f6-940f-4654-a666-0d800a3c31fd-local-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:47:15.466338 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:15.466327    4473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9db605f6-940f-4654-a666-0d800a3c31fd-istio-token\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:47:15.466338 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:15.466336    4473 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9db605f6-940f-4654-a666-0d800a3c31fd-istio-kubeconfig\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:47:15.466338 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:15.466346    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbqtq\" (UniqueName: \"kubernetes.io/projected/9db605f6-940f-4654-a666-0d800a3c31fd-kube-api-access-wbqtq\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:47:15.466683 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:15.466358    4473 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9db605f6-940f-4654-a666-0d800a3c31fd-cacerts\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:47:15.466683 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:15.466367    4473 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9db605f6-940f-4654-a666-0d800a3c31fd-istio-csr-dns-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:47:15.466683 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:15.466375    4473 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9db605f6-940f-4654-a666-0d800a3c31fd-istio-csr-ca-configmap\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:47:15.738910 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:15.738872    4473 generic.go:356] "Generic (PLEG): container finished" podID="9db605f6-940f-4654-a666-0d800a3c31fd" containerID="4a1af2706ab0fccfa373d52a77c139c6fafb67c5b635e85e0fb99cd24c6b0e54" exitCode=0
Nov 03 08:47:15.739470 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:15.738932    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv"
Nov 03 08:47:15.739470 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:15.738986    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv" event={"ID":"9db605f6-940f-4654-a666-0d800a3c31fd","Type":"ContainerDied","Data":"4a1af2706ab0fccfa373d52a77c139c6fafb67c5b635e85e0fb99cd24c6b0e54"}
Nov 03 08:47:15.739470 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:15.739044    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv" event={"ID":"9db605f6-940f-4654-a666-0d800a3c31fd","Type":"ContainerDied","Data":"a8209a19ff98a6f27a91a9b0802fbcae35fa8d561c11a34006469313a9ce8981"}
Nov 03 08:47:15.739470 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:15.739076    4473 scope.go:122] "RemoveContainer" containerID="4a1af2706ab0fccfa373d52a77c139c6fafb67c5b635e85e0fb99cd24c6b0e54"
Nov 03 08:47:15.752263 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:15.752244    4473 scope.go:122] "RemoveContainer" containerID="4a1af2706ab0fccfa373d52a77c139c6fafb67c5b635e85e0fb99cd24c6b0e54"
Nov 03 08:47:15.752576 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:47:15.752551    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a1af2706ab0fccfa373d52a77c139c6fafb67c5b635e85e0fb99cd24c6b0e54\": container with ID starting with 4a1af2706ab0fccfa373d52a77c139c6fafb67c5b635e85e0fb99cd24c6b0e54 not found: ID does not exist" containerID="4a1af2706ab0fccfa373d52a77c139c6fafb67c5b635e85e0fb99cd24c6b0e54"
Nov 03 08:47:15.752658 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:15.752589    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a1af2706ab0fccfa373d52a77c139c6fafb67c5b635e85e0fb99cd24c6b0e54"} err="failed to get container status \"4a1af2706ab0fccfa373d52a77c139c6fafb67c5b635e85e0fb99cd24c6b0e54\": rpc error: code = NotFound desc = could not find container \"4a1af2706ab0fccfa373d52a77c139c6fafb67c5b635e85e0fb99cd24c6b0e54\": container with ID starting with 4a1af2706ab0fccfa373d52a77c139c6fafb67c5b635e85e0fb99cd24c6b0e54 not found: ID does not exist"
Nov 03 08:47:15.908119 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:15.908090    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv"]
Nov 03 08:47:15.959177 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:15.959143    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-4bwtv"]
Nov 03 08:47:16.938248 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:47:16.938218    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:47:17.750008 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:17.749972    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p72nd" event={"ID":"dc80f365-2a40-4fd0-ae2b-6cfc815a3d63","Type":"ContainerStarted","Data":"8c92f6dae5ddd7e7859de899cc3c5a186368645ecdaa222b3da885e9276e0ebc"}
Nov 03 08:47:17.750177 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:17.750107    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p72nd"
Nov 03 08:47:17.868551 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:17.868523    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9db605f6-940f-4654-a666-0d800a3c31fd" path="/var/lib/kubelet/pods/9db605f6-940f-4654-a666-0d800a3c31fd/volumes"
Nov 03 08:47:17.876302 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:17.876246    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p72nd" podStartSLOduration=3.198634181 podStartE2EDuration="15.876229517s" totalImagesPullingTime="12.677595336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:47:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:47:17.875769309 +0000 UTC m=+462.144832754" watchObservedRunningTime="2026-11-03 08:47:17.876229517 +0000 UTC m=+462.145292953"
Nov 03 08:47:18.027817 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:18.027739    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4vkwx"]
Nov 03 08:47:18.028174 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:18.028086    4473 cpu_manager.go:414] "containerMap: removing container" podUID="9db605f6-940f-4654-a666-0d800a3c31fd" containerName="discovery"
Nov 03 08:47:18.028174 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:18.028096    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9db605f6-940f-4654-a666-0d800a3c31fd" containerName="discovery"
Nov 03 08:47:18.028174 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:18.028161    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="9db605f6-940f-4654-a666-0d800a3c31fd" containerName="discovery"
Nov 03 08:47:18.032172 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:18.032152    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4vkwx"
Nov 03 08:47:18.089697 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:18.089390    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-jhmbq\""
Nov 03 08:47:18.089697 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:18.089506    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Nov 03 08:47:18.089914 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:18.089774    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4vkwx"]
Nov 03 08:47:18.188113 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:18.188078    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e69d110-2e0a-4cfd-8cf7-15df1299374b-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-4vkwx\" (UID: \"2e69d110-2e0a-4cfd-8cf7-15df1299374b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4vkwx"
Nov 03 08:47:18.188280 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:18.188152    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rbbh\" (UniqueName: \"kubernetes.io/projected/2e69d110-2e0a-4cfd-8cf7-15df1299374b-kube-api-access-4rbbh\") pod \"llmisvc-controller-manager-869d5dcc5d-4vkwx\" (UID: \"2e69d110-2e0a-4cfd-8cf7-15df1299374b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4vkwx"
Nov 03 08:47:18.288821 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:18.288741    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rbbh\" (UniqueName: \"kubernetes.io/projected/2e69d110-2e0a-4cfd-8cf7-15df1299374b-kube-api-access-4rbbh\") pod \"llmisvc-controller-manager-869d5dcc5d-4vkwx\" (UID: \"2e69d110-2e0a-4cfd-8cf7-15df1299374b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4vkwx"
Nov 03 08:47:18.288821 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:18.288789    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e69d110-2e0a-4cfd-8cf7-15df1299374b-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-4vkwx\" (UID: \"2e69d110-2e0a-4cfd-8cf7-15df1299374b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4vkwx"
Nov 03 08:47:18.291330 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:18.291311    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e69d110-2e0a-4cfd-8cf7-15df1299374b-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-4vkwx\" (UID: \"2e69d110-2e0a-4cfd-8cf7-15df1299374b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4vkwx"
Nov 03 08:47:18.349652 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:18.349630    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rbbh\" (UniqueName: \"kubernetes.io/projected/2e69d110-2e0a-4cfd-8cf7-15df1299374b-kube-api-access-4rbbh\") pod \"llmisvc-controller-manager-869d5dcc5d-4vkwx\" (UID: \"2e69d110-2e0a-4cfd-8cf7-15df1299374b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4vkwx"
Nov 03 08:47:18.644447 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:18.644419    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4vkwx"
Nov 03 08:47:19.018337 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:19.018314    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4vkwx"]
Nov 03 08:47:19.020046 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:47:19.020015    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2e69d110_2e0a_4cfd_8cf7_15df1299374b.slice/crio-cd98cdc7343634f040a9f2e5d733594c88bb631bc80c62ec3e7b461fe95d9478 WatchSource:0}: Error finding container cd98cdc7343634f040a9f2e5d733594c88bb631bc80c62ec3e7b461fe95d9478: Status 404 returned error can't find the container with id cd98cdc7343634f040a9f2e5d733594c88bb631bc80c62ec3e7b461fe95d9478
Nov 03 08:47:19.760228 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:19.760196    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4vkwx" event={"ID":"2e69d110-2e0a-4cfd-8cf7-15df1299374b","Type":"ContainerStarted","Data":"cd98cdc7343634f040a9f2e5d733594c88bb631bc80c62ec3e7b461fe95d9478"}
Nov 03 08:47:21.770928 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:21.770885    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4vkwx" event={"ID":"2e69d110-2e0a-4cfd-8cf7-15df1299374b","Type":"ContainerStarted","Data":"5f46f645552877d72236960fbe8884a9f8ee2112d64e423095009a1497880a15"}
Nov 03 08:47:21.771316 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:21.771024    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4vkwx"
Nov 03 08:47:21.893593 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:21.893544    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4vkwx" podStartSLOduration=2.647077657 podStartE2EDuration="4.893531697s" totalImagesPullingTime="2.24645404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:47:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:47:21.892375578 +0000 UTC m=+466.161439011" watchObservedRunningTime="2026-11-03 08:47:21.893531697 +0000 UTC m=+466.162595130"
Nov 03 08:47:28.760876 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:28.760845    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-p72nd"
Nov 03 08:47:39.350999 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:39.350963    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 08:47:46.936184 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:47:46.936105    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:47:52.777611 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:47:52.777560    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4vkwx"
Nov 03 08:48:16.936153 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:48:16.936118    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:48:33.661773 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:48:33.661742    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-6vvpk"]
Nov 03 08:48:33.663901 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:48:33.663885    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-6vvpk"
Nov 03 08:48:33.729639 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:48:33.729598    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v79b\" (UniqueName: \"kubernetes.io/projected/73e8a24d-31cf-44f0-99a5-5d3bd7259449-kube-api-access-5v79b\") pod \"llmisvc-controller-manager-6466c554d8-6vvpk\" (UID: \"73e8a24d-31cf-44f0-99a5-5d3bd7259449\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-6vvpk"
Nov 03 08:48:33.729639 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:48:33.729637    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73e8a24d-31cf-44f0-99a5-5d3bd7259449-cert\") pod \"llmisvc-controller-manager-6466c554d8-6vvpk\" (UID: \"73e8a24d-31cf-44f0-99a5-5d3bd7259449\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-6vvpk"
Nov 03 08:48:33.730615 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:48:33.730589    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-6vvpk"]
Nov 03 08:48:33.830667 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:48:33.830631    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v79b\" (UniqueName: \"kubernetes.io/projected/73e8a24d-31cf-44f0-99a5-5d3bd7259449-kube-api-access-5v79b\") pod \"llmisvc-controller-manager-6466c554d8-6vvpk\" (UID: \"73e8a24d-31cf-44f0-99a5-5d3bd7259449\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-6vvpk"
Nov 03 08:48:33.830667 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:48:33.830674    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73e8a24d-31cf-44f0-99a5-5d3bd7259449-cert\") pod \"llmisvc-controller-manager-6466c554d8-6vvpk\" (UID: \"73e8a24d-31cf-44f0-99a5-5d3bd7259449\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-6vvpk"
Nov 03 08:48:33.833307 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:48:33.833281    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73e8a24d-31cf-44f0-99a5-5d3bd7259449-cert\") pod \"llmisvc-controller-manager-6466c554d8-6vvpk\" (UID: \"73e8a24d-31cf-44f0-99a5-5d3bd7259449\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-6vvpk"
Nov 03 08:48:33.890488 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:48:33.890456    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v79b\" (UniqueName: \"kubernetes.io/projected/73e8a24d-31cf-44f0-99a5-5d3bd7259449-kube-api-access-5v79b\") pod \"llmisvc-controller-manager-6466c554d8-6vvpk\" (UID: \"73e8a24d-31cf-44f0-99a5-5d3bd7259449\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-6vvpk"
Nov 03 08:48:33.975297 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:48:33.975221    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-6vvpk"
Nov 03 08:48:34.347059 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:48:34.347037    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-6vvpk"]
Nov 03 08:48:34.349572 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:48:34.349548    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e8a24d_31cf_44f0_99a5_5d3bd7259449.slice/crio-8550c4ac3ed2fe8f10895c1131327f395bebe1d140290be437b0b4ace858523e WatchSource:0}: Error finding container 8550c4ac3ed2fe8f10895c1131327f395bebe1d140290be437b0b4ace858523e: Status 404 returned error can't find the container with id 8550c4ac3ed2fe8f10895c1131327f395bebe1d140290be437b0b4ace858523e
Nov 03 08:48:35.083490 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:48:35.083458    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-6vvpk" event={"ID":"73e8a24d-31cf-44f0-99a5-5d3bd7259449","Type":"ContainerStarted","Data":"8550c4ac3ed2fe8f10895c1131327f395bebe1d140290be437b0b4ace858523e"}
Nov 03 08:48:36.096047 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:48:36.095415    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-6vvpk" event={"ID":"73e8a24d-31cf-44f0-99a5-5d3bd7259449","Type":"ContainerStarted","Data":"b7a8cf812fcaf2b294d559b284cfe90da9358bf4f7ebc46af42ff327ca665e5a"}
Nov 03 08:48:36.096047 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:48:36.095973    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-6vvpk"
Nov 03 08:48:36.213537 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:48:36.213483    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-6vvpk" podStartSLOduration=2.415607254 podStartE2EDuration="3.213468473s" totalImagesPullingTime="797.861219ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:48:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:48:36.212069568 +0000 UTC m=+540.481133035" watchObservedRunningTime="2026-11-03 08:48:36.213468473 +0000 UTC m=+540.482531924"
Nov 03 08:48:46.937336 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:48:46.937296    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:49:07.101715 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:49:07.101685    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-6vvpk"
Nov 03 08:49:07.270418 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:49:07.270373    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4vkwx"]
Nov 03 08:49:07.270648 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:49:07.270612    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4vkwx" podUID="2e69d110-2e0a-4cfd-8cf7-15df1299374b" containerName="manager" containerID="cri-o://5f46f645552877d72236960fbe8884a9f8ee2112d64e423095009a1497880a15" gracePeriod=30
Nov 03 08:49:07.506015 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:49:07.505990    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4vkwx"
Nov 03 08:49:07.628587 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:49:07.628559    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e69d110-2e0a-4cfd-8cf7-15df1299374b-cert\" (UniqueName: \"kubernetes.io/secret/2e69d110-2e0a-4cfd-8cf7-15df1299374b-cert\") pod \"2e69d110-2e0a-4cfd-8cf7-15df1299374b\" (UID: \"2e69d110-2e0a-4cfd-8cf7-15df1299374b\") "
Nov 03 08:49:07.628781 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:49:07.628600    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e69d110-2e0a-4cfd-8cf7-15df1299374b-kube-api-access-4rbbh\" (UniqueName: \"kubernetes.io/projected/2e69d110-2e0a-4cfd-8cf7-15df1299374b-kube-api-access-4rbbh\") pod \"2e69d110-2e0a-4cfd-8cf7-15df1299374b\" (UID: \"2e69d110-2e0a-4cfd-8cf7-15df1299374b\") "
Nov 03 08:49:07.631308 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:49:07.631274    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e69d110-2e0a-4cfd-8cf7-15df1299374b-cert" pod "2e69d110-2e0a-4cfd-8cf7-15df1299374b" (UID: "2e69d110-2e0a-4cfd-8cf7-15df1299374b"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:49:07.631468 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:49:07.631444    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e69d110-2e0a-4cfd-8cf7-15df1299374b-kube-api-access-4rbbh" pod "2e69d110-2e0a-4cfd-8cf7-15df1299374b" (UID: "2e69d110-2e0a-4cfd-8cf7-15df1299374b"). InnerVolumeSpecName "kube-api-access-4rbbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:49:07.729747 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:49:07.729664    4473 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e69d110-2e0a-4cfd-8cf7-15df1299374b-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:49:07.729747 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:49:07.729694    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rbbh\" (UniqueName: \"kubernetes.io/projected/2e69d110-2e0a-4cfd-8cf7-15df1299374b-kube-api-access-4rbbh\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:49:08.235377 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:49:08.235345    4473 generic.go:356] "Generic (PLEG): container finished" podID="2e69d110-2e0a-4cfd-8cf7-15df1299374b" containerID="5f46f645552877d72236960fbe8884a9f8ee2112d64e423095009a1497880a15" exitCode=0
Nov 03 08:49:08.235785 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:49:08.235381    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4vkwx" event={"ID":"2e69d110-2e0a-4cfd-8cf7-15df1299374b","Type":"ContainerDied","Data":"5f46f645552877d72236960fbe8884a9f8ee2112d64e423095009a1497880a15"}
Nov 03 08:49:08.235785 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:49:08.235401    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4vkwx"
Nov 03 08:49:08.235785 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:49:08.235446    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4vkwx" event={"ID":"2e69d110-2e0a-4cfd-8cf7-15df1299374b","Type":"ContainerDied","Data":"cd98cdc7343634f040a9f2e5d733594c88bb631bc80c62ec3e7b461fe95d9478"}
Nov 03 08:49:08.235785 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:49:08.235469    4473 scope.go:122] "RemoveContainer" containerID="5f46f645552877d72236960fbe8884a9f8ee2112d64e423095009a1497880a15"
Nov 03 08:49:08.243921 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:49:08.243905    4473 scope.go:122] "RemoveContainer" containerID="5f46f645552877d72236960fbe8884a9f8ee2112d64e423095009a1497880a15"
Nov 03 08:49:08.244214 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:49:08.244190    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f46f645552877d72236960fbe8884a9f8ee2112d64e423095009a1497880a15\": container with ID starting with 5f46f645552877d72236960fbe8884a9f8ee2112d64e423095009a1497880a15 not found: ID does not exist" containerID="5f46f645552877d72236960fbe8884a9f8ee2112d64e423095009a1497880a15"
Nov 03 08:49:08.244294 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:49:08.244222    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f46f645552877d72236960fbe8884a9f8ee2112d64e423095009a1497880a15"} err="failed to get container status \"5f46f645552877d72236960fbe8884a9f8ee2112d64e423095009a1497880a15\": rpc error: code = NotFound desc = could not find container \"5f46f645552877d72236960fbe8884a9f8ee2112d64e423095009a1497880a15\": container with ID starting with 5f46f645552877d72236960fbe8884a9f8ee2112d64e423095009a1497880a15 not found: ID does not exist"
Nov 03 08:49:08.404212 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:49:08.404182    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4vkwx"]
Nov 03 08:49:08.411225 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:49:08.411202    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4vkwx"]
Nov 03 08:49:09.868097 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:49:09.868065    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e69d110-2e0a-4cfd-8cf7-15df1299374b" path="/var/lib/kubelet/pods/2e69d110-2e0a-4cfd-8cf7-15df1299374b/volumes"
Nov 03 08:49:16.935872 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:49:16.935751    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:49:28.320149 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:49:28.320065    4473 generic.go:356] "Generic (PLEG): container finished" podID="17e69a7f-4e64-4aad-b798-af8fa74a2b76" containerID="367fb63de7da5a3a8d40ca0f481177808f079ee612f30457884b3c459ae49741" exitCode=1
Nov 03 08:49:28.320503 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:49:28.320139    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dd5f74f7b-lk7c9" event={"ID":"17e69a7f-4e64-4aad-b798-af8fa74a2b76","Type":"ContainerDied","Data":"367fb63de7da5a3a8d40ca0f481177808f079ee612f30457884b3c459ae49741"}
Nov 03 08:49:28.320503 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:49:28.320489    4473 scope.go:122] "RemoveContainer" containerID="367fb63de7da5a3a8d40ca0f481177808f079ee612f30457884b3c459ae49741"
Nov 03 08:49:29.326420 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:49:29.326389    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dd5f74f7b-lk7c9" event={"ID":"17e69a7f-4e64-4aad-b798-af8fa74a2b76","Type":"ContainerStarted","Data":"339a02004ef32a7c64014efdcd89a49bf9a2b6f267e85d31d748f42e98b1ffa4"}
Nov 03 08:49:29.326843 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:49:29.326576    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7dd5f74f7b-lk7c9"
Nov 03 08:49:35.928824 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:49:35.928795    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 08:49:35.930493 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:49:35.930468    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 08:49:40.333199 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:49:40.333171    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7dd5f74f7b-lk7c9"
Nov 03 08:49:46.940363 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:49:46.940325    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:49:52.089099 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:49:52.089061    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 08:50:02.432969 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:02.432910    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 08:50:11.454414 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:11.454381    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 08:50:16.935829 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:50:16.935793    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:50:20.585934 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:20.585904    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 08:50:32.283567 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:32.283534    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 08:50:32.609049 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:32.608971    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"]
Nov 03 08:50:32.609339 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:32.609326    4473 cpu_manager.go:414] "containerMap: removing container" podUID="2e69d110-2e0a-4cfd-8cf7-15df1299374b" containerName="manager"
Nov 03 08:50:32.609390 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:32.609340    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e69d110-2e0a-4cfd-8cf7-15df1299374b" containerName="manager"
Nov 03 08:50:32.609433 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:32.609392    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e69d110-2e0a-4cfd-8cf7-15df1299374b" containerName="manager"
Nov 03 08:50:32.612335 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:32.612321    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"
Nov 03 08:50:32.665478 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:32.665451    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-pqxg4\""
Nov 03 08:50:32.718472 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:32.718442    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"]
Nov 03 08:50:32.739825 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:32.739796    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14dd6c95-a250-41c5-87c7-ae741f96d01a-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whmmw\" (UID: \"14dd6c95-a250-41c5-87c7-ae741f96d01a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"
Nov 03 08:50:32.739825 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:32.739828    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzp75\" (UniqueName: \"kubernetes.io/projected/14dd6c95-a250-41c5-87c7-ae741f96d01a-kube-api-access-qzp75\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whmmw\" (UID: \"14dd6c95-a250-41c5-87c7-ae741f96d01a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"
Nov 03 08:50:32.740015 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:32.739906    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14dd6c95-a250-41c5-87c7-ae741f96d01a-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whmmw\" (UID: \"14dd6c95-a250-41c5-87c7-ae741f96d01a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"
Nov 03 08:50:32.740015 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:32.739966    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14dd6c95-a250-41c5-87c7-ae741f96d01a-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whmmw\" (UID: \"14dd6c95-a250-41c5-87c7-ae741f96d01a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"
Nov 03 08:50:32.740015 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:32.740000    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14dd6c95-a250-41c5-87c7-ae741f96d01a-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whmmw\" (UID: \"14dd6c95-a250-41c5-87c7-ae741f96d01a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"
Nov 03 08:50:32.740115 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:32.740023    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14dd6c95-a250-41c5-87c7-ae741f96d01a-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whmmw\" (UID: \"14dd6c95-a250-41c5-87c7-ae741f96d01a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"
Nov 03 08:50:32.740147 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:32.740121    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14dd6c95-a250-41c5-87c7-ae741f96d01a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whmmw\" (UID: \"14dd6c95-a250-41c5-87c7-ae741f96d01a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"
Nov 03 08:50:32.740180 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:32.740156    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14dd6c95-a250-41c5-87c7-ae741f96d01a-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whmmw\" (UID: \"14dd6c95-a250-41c5-87c7-ae741f96d01a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"
Nov 03 08:50:32.740215 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:32.740181    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14dd6c95-a250-41c5-87c7-ae741f96d01a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whmmw\" (UID: \"14dd6c95-a250-41c5-87c7-ae741f96d01a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"
Nov 03 08:50:32.841016 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:32.840983    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14dd6c95-a250-41c5-87c7-ae741f96d01a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whmmw\" (UID: \"14dd6c95-a250-41c5-87c7-ae741f96d01a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"
Nov 03 08:50:32.841016 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:32.841022    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14dd6c95-a250-41c5-87c7-ae741f96d01a-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whmmw\" (UID: \"14dd6c95-a250-41c5-87c7-ae741f96d01a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"
Nov 03 08:50:32.841226 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:32.841054    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14dd6c95-a250-41c5-87c7-ae741f96d01a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whmmw\" (UID: \"14dd6c95-a250-41c5-87c7-ae741f96d01a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"
Nov 03 08:50:32.841226 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:32.841079    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14dd6c95-a250-41c5-87c7-ae741f96d01a-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whmmw\" (UID: \"14dd6c95-a250-41c5-87c7-ae741f96d01a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"
Nov 03 08:50:32.841226 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:32.841102    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzp75\" (UniqueName: \"kubernetes.io/projected/14dd6c95-a250-41c5-87c7-ae741f96d01a-kube-api-access-qzp75\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whmmw\" (UID: \"14dd6c95-a250-41c5-87c7-ae741f96d01a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"
Nov 03 08:50:32.841383 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:32.841261    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14dd6c95-a250-41c5-87c7-ae741f96d01a-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whmmw\" (UID: \"14dd6c95-a250-41c5-87c7-ae741f96d01a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"
Nov 03 08:50:32.841383 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:32.841344    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14dd6c95-a250-41c5-87c7-ae741f96d01a-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whmmw\" (UID: \"14dd6c95-a250-41c5-87c7-ae741f96d01a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"
Nov 03 08:50:32.841484 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:32.841397    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14dd6c95-a250-41c5-87c7-ae741f96d01a-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whmmw\" (UID: \"14dd6c95-a250-41c5-87c7-ae741f96d01a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"
Nov 03 08:50:32.841484 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:32.841426    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14dd6c95-a250-41c5-87c7-ae741f96d01a-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whmmw\" (UID: \"14dd6c95-a250-41c5-87c7-ae741f96d01a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"
Nov 03 08:50:32.841484 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:32.841440    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14dd6c95-a250-41c5-87c7-ae741f96d01a-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whmmw\" (UID: \"14dd6c95-a250-41c5-87c7-ae741f96d01a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"
Nov 03 08:50:32.841651 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:32.841630    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14dd6c95-a250-41c5-87c7-ae741f96d01a-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whmmw\" (UID: \"14dd6c95-a250-41c5-87c7-ae741f96d01a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"
Nov 03 08:50:32.841738 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:32.841714    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14dd6c95-a250-41c5-87c7-ae741f96d01a-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whmmw\" (UID: \"14dd6c95-a250-41c5-87c7-ae741f96d01a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"
Nov 03 08:50:32.842149 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:32.841836    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14dd6c95-a250-41c5-87c7-ae741f96d01a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whmmw\" (UID: \"14dd6c95-a250-41c5-87c7-ae741f96d01a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"
Nov 03 08:50:32.842149 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:32.841851    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14dd6c95-a250-41c5-87c7-ae741f96d01a-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whmmw\" (UID: \"14dd6c95-a250-41c5-87c7-ae741f96d01a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"
Nov 03 08:50:32.843702 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:32.843680    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14dd6c95-a250-41c5-87c7-ae741f96d01a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whmmw\" (UID: \"14dd6c95-a250-41c5-87c7-ae741f96d01a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"
Nov 03 08:50:32.843809 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:32.843765    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14dd6c95-a250-41c5-87c7-ae741f96d01a-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whmmw\" (UID: \"14dd6c95-a250-41c5-87c7-ae741f96d01a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"
Nov 03 08:50:32.900727 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:32.900686    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14dd6c95-a250-41c5-87c7-ae741f96d01a-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whmmw\" (UID: \"14dd6c95-a250-41c5-87c7-ae741f96d01a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"
Nov 03 08:50:32.901020 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:32.900998    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzp75\" (UniqueName: \"kubernetes.io/projected/14dd6c95-a250-41c5-87c7-ae741f96d01a-kube-api-access-qzp75\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-whmmw\" (UID: \"14dd6c95-a250-41c5-87c7-ae741f96d01a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"
Nov 03 08:50:32.924614 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:32.924596    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"
Nov 03 08:50:33.350874 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:33.350850    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"]
Nov 03 08:50:33.352866 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:50:33.352839    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14dd6c95_a250_41c5_87c7_ae741f96d01a.slice/crio-90288be7bcdc1fedf6db7681904e923254f14d08434e03cd40861fbbb64f3559 WatchSource:0}: Error finding container 90288be7bcdc1fedf6db7681904e923254f14d08434e03cd40861fbbb64f3559: Status 404 returned error can't find the container with id 90288be7bcdc1fedf6db7681904e923254f14d08434e03cd40861fbbb64f3559
Nov 03 08:50:33.354832 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:33.354817    4473 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 08:50:33.607891 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:33.607820    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw" event={"ID":"14dd6c95-a250-41c5-87c7-ae741f96d01a","Type":"ContainerStarted","Data":"90288be7bcdc1fedf6db7681904e923254f14d08434e03cd40861fbbb64f3559"}
Nov 03 08:50:37.310386 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:37.310350    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 08:50:37.310683 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:37.310421    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 08:50:37.310683 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:37.310448    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 08:50:37.630246 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:37.630210    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw" event={"ID":"14dd6c95-a250-41c5-87c7-ae741f96d01a","Type":"ContainerStarted","Data":"6a3e68602df23f6d3e0fa7b142c264ab2419d360791c94438dbafdf331faf95e"}
Nov 03 08:50:37.801178 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:37.801122    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw" podStartSLOduration=1.846578179 podStartE2EDuration="5.80110895s" totalImagesPullingTime="3.954530771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:50:37.799838691 +0000 UTC m=+662.068902148" watchObservedRunningTime="2026-11-03 08:50:37.80110895 +0000 UTC m=+662.070172385"
Nov 03 08:50:37.927032 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:37.926925    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"
Nov 03 08:50:38.936970 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:38.933391    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-whmmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 03 08:50:38.936970 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:38.933448    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw" podUID="14dd6c95-a250-41c5-87c7-ae741f96d01a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 03 08:50:39.933406 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:39.933368    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-whmmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 03 08:50:39.933704 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:39.933675    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw" podUID="14dd6c95-a250-41c5-87c7-ae741f96d01a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 03 08:50:40.928971 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:40.928036    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-whmmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 03 08:50:40.928971 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:40.928095    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw" podUID="14dd6c95-a250-41c5-87c7-ae741f96d01a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 03 08:50:41.941095 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:41.941059    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-whmmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 03 08:50:41.941480 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:41.941114    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw" podUID="14dd6c95-a250-41c5-87c7-ae741f96d01a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 03 08:50:42.933973 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:42.927293    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-whmmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 03 08:50:42.933973 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:42.927366    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw" podUID="14dd6c95-a250-41c5-87c7-ae741f96d01a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 03 08:50:43.925156 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:43.925121    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-whmmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 03 08:50:43.925557 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:43.925178    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw" podUID="14dd6c95-a250-41c5-87c7-ae741f96d01a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 03 08:50:45.675567 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:45.675535    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"
Nov 03 08:50:45.676158 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:45.676131    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"
Nov 03 08:50:45.677142 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:45.677123    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"
Nov 03 08:50:46.936263 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:50:46.936188    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:50:47.608819 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:47.608788    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"]
Nov 03 08:50:47.612277 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:47.612255    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"
Nov 03 08:50:47.797702 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:47.797672    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94998825-9ad4-4f71-9c54-b4c2a5032c26-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c6bh2\" (UID: \"94998825-9ad4-4f71-9c54-b4c2a5032c26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"
Nov 03 08:50:47.797873 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:47.797717    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94998825-9ad4-4f71-9c54-b4c2a5032c26-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c6bh2\" (UID: \"94998825-9ad4-4f71-9c54-b4c2a5032c26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"
Nov 03 08:50:47.797873 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:47.797741    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94998825-9ad4-4f71-9c54-b4c2a5032c26-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c6bh2\" (UID: \"94998825-9ad4-4f71-9c54-b4c2a5032c26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"
Nov 03 08:50:47.797873 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:47.797792    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjjtm\" (UniqueName: \"kubernetes.io/projected/94998825-9ad4-4f71-9c54-b4c2a5032c26-kube-api-access-jjjtm\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c6bh2\" (UID: \"94998825-9ad4-4f71-9c54-b4c2a5032c26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"
Nov 03 08:50:47.798066 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:47.797971    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94998825-9ad4-4f71-9c54-b4c2a5032c26-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c6bh2\" (UID: \"94998825-9ad4-4f71-9c54-b4c2a5032c26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"
Nov 03 08:50:47.798066 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:47.798041    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94998825-9ad4-4f71-9c54-b4c2a5032c26-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c6bh2\" (UID: \"94998825-9ad4-4f71-9c54-b4c2a5032c26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"
Nov 03 08:50:47.798184 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:47.798169    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94998825-9ad4-4f71-9c54-b4c2a5032c26-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c6bh2\" (UID: \"94998825-9ad4-4f71-9c54-b4c2a5032c26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"
Nov 03 08:50:47.798242 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:47.798200    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94998825-9ad4-4f71-9c54-b4c2a5032c26-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c6bh2\" (UID: \"94998825-9ad4-4f71-9c54-b4c2a5032c26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"
Nov 03 08:50:47.798356 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:47.798330    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94998825-9ad4-4f71-9c54-b4c2a5032c26-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c6bh2\" (UID: \"94998825-9ad4-4f71-9c54-b4c2a5032c26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"
Nov 03 08:50:47.834300 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:47.834276    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"]
Nov 03 08:50:47.898726 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:47.898701    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94998825-9ad4-4f71-9c54-b4c2a5032c26-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c6bh2\" (UID: \"94998825-9ad4-4f71-9c54-b4c2a5032c26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"
Nov 03 08:50:47.898886 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:47.898731    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94998825-9ad4-4f71-9c54-b4c2a5032c26-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c6bh2\" (UID: \"94998825-9ad4-4f71-9c54-b4c2a5032c26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"
Nov 03 08:50:47.898886 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:47.898753    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94998825-9ad4-4f71-9c54-b4c2a5032c26-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c6bh2\" (UID: \"94998825-9ad4-4f71-9c54-b4c2a5032c26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"
Nov 03 08:50:47.899006 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:47.898880    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94998825-9ad4-4f71-9c54-b4c2a5032c26-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c6bh2\" (UID: \"94998825-9ad4-4f71-9c54-b4c2a5032c26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"
Nov 03 08:50:47.899006 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:47.898966    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94998825-9ad4-4f71-9c54-b4c2a5032c26-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c6bh2\" (UID: \"94998825-9ad4-4f71-9c54-b4c2a5032c26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"
Nov 03 08:50:47.899117 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:47.899012    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94998825-9ad4-4f71-9c54-b4c2a5032c26-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c6bh2\" (UID: \"94998825-9ad4-4f71-9c54-b4c2a5032c26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"
Nov 03 08:50:47.899117 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:47.899042    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjjtm\" (UniqueName: \"kubernetes.io/projected/94998825-9ad4-4f71-9c54-b4c2a5032c26-kube-api-access-jjjtm\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c6bh2\" (UID: \"94998825-9ad4-4f71-9c54-b4c2a5032c26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"
Nov 03 08:50:47.899117 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:47.899084    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94998825-9ad4-4f71-9c54-b4c2a5032c26-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c6bh2\" (UID: \"94998825-9ad4-4f71-9c54-b4c2a5032c26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"
Nov 03 08:50:47.899117 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:47.899116    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94998825-9ad4-4f71-9c54-b4c2a5032c26-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c6bh2\" (UID: \"94998825-9ad4-4f71-9c54-b4c2a5032c26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"
Nov 03 08:50:47.899392 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:47.899358    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94998825-9ad4-4f71-9c54-b4c2a5032c26-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c6bh2\" (UID: \"94998825-9ad4-4f71-9c54-b4c2a5032c26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"
Nov 03 08:50:47.899692 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:47.899473    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94998825-9ad4-4f71-9c54-b4c2a5032c26-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c6bh2\" (UID: \"94998825-9ad4-4f71-9c54-b4c2a5032c26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"
Nov 03 08:50:47.899692 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:47.899552    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94998825-9ad4-4f71-9c54-b4c2a5032c26-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c6bh2\" (UID: \"94998825-9ad4-4f71-9c54-b4c2a5032c26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"
Nov 03 08:50:47.899832 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:47.899686    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94998825-9ad4-4f71-9c54-b4c2a5032c26-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c6bh2\" (UID: \"94998825-9ad4-4f71-9c54-b4c2a5032c26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"
Nov 03 08:50:47.899832 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:47.899714    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94998825-9ad4-4f71-9c54-b4c2a5032c26-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c6bh2\" (UID: \"94998825-9ad4-4f71-9c54-b4c2a5032c26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"
Nov 03 08:50:47.901222 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:47.901199    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94998825-9ad4-4f71-9c54-b4c2a5032c26-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c6bh2\" (UID: \"94998825-9ad4-4f71-9c54-b4c2a5032c26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"
Nov 03 08:50:47.902496 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:47.902474    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94998825-9ad4-4f71-9c54-b4c2a5032c26-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c6bh2\" (UID: \"94998825-9ad4-4f71-9c54-b4c2a5032c26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"
Nov 03 08:50:47.958584 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:47.958556    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94998825-9ad4-4f71-9c54-b4c2a5032c26-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c6bh2\" (UID: \"94998825-9ad4-4f71-9c54-b4c2a5032c26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"
Nov 03 08:50:47.958891 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:47.958715    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjjtm\" (UniqueName: \"kubernetes.io/projected/94998825-9ad4-4f71-9c54-b4c2a5032c26-kube-api-access-jjjtm\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c6bh2\" (UID: \"94998825-9ad4-4f71-9c54-b4c2a5032c26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"
Nov 03 08:50:48.224600 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:48.224519    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"
Nov 03 08:50:48.654122 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:48.654095    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"]
Nov 03 08:50:48.657291 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:50:48.657262    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94998825_9ad4_4f71_9c54_b4c2a5032c26.slice/crio-e4b7fb1df61d224a6f3653a276f7c97e61eb01bc7458ef3de4021397c617547e WatchSource:0}: Error finding container e4b7fb1df61d224a6f3653a276f7c97e61eb01bc7458ef3de4021397c617547e: Status 404 returned error can't find the container with id e4b7fb1df61d224a6f3653a276f7c97e61eb01bc7458ef3de4021397c617547e
Nov 03 08:50:48.660211 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:48.660179    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 08:50:48.660277 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:48.660247    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 08:50:48.660314 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:48.660278    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 08:50:48.690210 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:48.690177    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2" event={"ID":"94998825-9ad4-4f71-9c54-b4c2a5032c26","Type":"ContainerStarted","Data":"e4b7fb1df61d224a6f3653a276f7c97e61eb01bc7458ef3de4021397c617547e"}
Nov 03 08:50:49.697458 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:49.697418    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2" event={"ID":"94998825-9ad4-4f71-9c54-b4c2a5032c26","Type":"ContainerStarted","Data":"0b56a3c4a3f67bd48d72134dcfb0a424cc3a3f209541161ad7650c737001076a"}
Nov 03 08:50:49.943313 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:49.943246    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2" podStartSLOduration=2.943224104 podStartE2EDuration="2.943224104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:50:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:50:49.869536606 +0000 UTC m=+674.138600062" watchObservedRunningTime="2026-11-03 08:50:49.943224104 +0000 UTC m=+674.212287540"
Nov 03 08:50:50.239971 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:50.239244    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"
Nov 03 08:50:50.242007 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:50.241981    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 03 08:50:50.242140 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:50.242037    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2" podUID="94998825-9ad4-4f71-9c54-b4c2a5032c26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 03 08:50:51.237972 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:51.230379    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 03 08:50:51.237972 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:51.230501    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2" podUID="94998825-9ad4-4f71-9c54-b4c2a5032c26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 03 08:50:52.232971 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:52.229348    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 03 08:50:52.232971 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:52.229425    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2" podUID="94998825-9ad4-4f71-9c54-b4c2a5032c26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 03 08:50:53.234310 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:53.234273    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 03 08:50:53.235255 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:53.234845    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2" podUID="94998825-9ad4-4f71-9c54-b4c2a5032c26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 03 08:50:54.228383 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:54.228349    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 03 08:50:54.228656 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:54.228626    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2" podUID="94998825-9ad4-4f71-9c54-b4c2a5032c26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 03 08:50:55.225192 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:55.225156    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 03 08:50:55.225584 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:55.225214    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2" podUID="94998825-9ad4-4f71-9c54-b4c2a5032c26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 03 08:50:56.937546 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:56.937511    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"
Nov 03 08:50:56.938272 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:56.938243    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"
Nov 03 08:50:56.939010 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:50:56.938991    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"
Nov 03 08:51:16.936748 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:51:16.936714    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:51:46.936806 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:51:46.936771    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:52:07.706169 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:07.706137    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6a1442a9-69f5d84b5f-hxfs9"]
Nov 03 08:52:07.709643 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:07.709628    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6a1442a9-69f5d84b5f-hxfs9"
Nov 03 08:52:07.767460 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:07.767419    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6a1442a9-69f5d84b5f-hxfs9"]
Nov 03 08:52:07.815479 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:07.815446    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6a1442a9\""
Nov 03 08:52:07.815645 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:07.815516    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6a1442a9-dockercfg-gzcbg\""
Nov 03 08:52:07.836726 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:07.836690    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a56b089-e9f1-454c-a5fd-32712c7f4119-plugins-config-volume\") pod \"payload-processing-e2e-ait-6a1442a9-69f5d84b5f-hxfs9\" (UID: \"3a56b089-e9f1-454c-a5fd-32712c7f4119\") " pod="openshift-ingress/payload-processing-e2e-ait-6a1442a9-69f5d84b5f-hxfs9"
Nov 03 08:52:07.836868 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:07.836767    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq686\" (UniqueName: \"kubernetes.io/projected/3a56b089-e9f1-454c-a5fd-32712c7f4119-kube-api-access-jq686\") pod \"payload-processing-e2e-ait-6a1442a9-69f5d84b5f-hxfs9\" (UID: \"3a56b089-e9f1-454c-a5fd-32712c7f4119\") " pod="openshift-ingress/payload-processing-e2e-ait-6a1442a9-69f5d84b5f-hxfs9"
Nov 03 08:52:07.937871 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:07.937834    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq686\" (UniqueName: \"kubernetes.io/projected/3a56b089-e9f1-454c-a5fd-32712c7f4119-kube-api-access-jq686\") pod \"payload-processing-e2e-ait-6a1442a9-69f5d84b5f-hxfs9\" (UID: \"3a56b089-e9f1-454c-a5fd-32712c7f4119\") " pod="openshift-ingress/payload-processing-e2e-ait-6a1442a9-69f5d84b5f-hxfs9"
Nov 03 08:52:07.938081 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:07.937962    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a56b089-e9f1-454c-a5fd-32712c7f4119-plugins-config-volume\") pod \"payload-processing-e2e-ait-6a1442a9-69f5d84b5f-hxfs9\" (UID: \"3a56b089-e9f1-454c-a5fd-32712c7f4119\") " pod="openshift-ingress/payload-processing-e2e-ait-6a1442a9-69f5d84b5f-hxfs9"
Nov 03 08:52:07.938683 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:07.938664    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a56b089-e9f1-454c-a5fd-32712c7f4119-plugins-config-volume\") pod \"payload-processing-e2e-ait-6a1442a9-69f5d84b5f-hxfs9\" (UID: \"3a56b089-e9f1-454c-a5fd-32712c7f4119\") " pod="openshift-ingress/payload-processing-e2e-ait-6a1442a9-69f5d84b5f-hxfs9"
Nov 03 08:52:07.999324 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:07.999264    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq686\" (UniqueName: \"kubernetes.io/projected/3a56b089-e9f1-454c-a5fd-32712c7f4119-kube-api-access-jq686\") pod \"payload-processing-e2e-ait-6a1442a9-69f5d84b5f-hxfs9\" (UID: \"3a56b089-e9f1-454c-a5fd-32712c7f4119\") " pod="openshift-ingress/payload-processing-e2e-ait-6a1442a9-69f5d84b5f-hxfs9"
Nov 03 08:52:08.022100 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:08.022064    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6a1442a9-69f5d84b5f-hxfs9"
Nov 03 08:52:08.393308 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:08.393280    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6a1442a9-69f5d84b5f-hxfs9"]
Nov 03 08:52:08.395775 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:52:08.395746    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a56b089_e9f1_454c_a5fd_32712c7f4119.slice/crio-b3f02738ed9e50da5c2b4bd5b66119e258e57abe98bd4399d28f841be9c0c866 WatchSource:0}: Error finding container b3f02738ed9e50da5c2b4bd5b66119e258e57abe98bd4399d28f841be9c0c866: Status 404 returned error can't find the container with id b3f02738ed9e50da5c2b4bd5b66119e258e57abe98bd4399d28f841be9c0c866
Nov 03 08:52:09.040259 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:09.040225    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6a1442a9-69f5d84b5f-hxfs9" event={"ID":"3a56b089-e9f1-454c-a5fd-32712c7f4119","Type":"ContainerStarted","Data":"b3f02738ed9e50da5c2b4bd5b66119e258e57abe98bd4399d28f841be9c0c866"}
Nov 03 08:52:11.051323 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:11.051293    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6a1442a9-69f5d84b5f-hxfs9" event={"ID":"3a56b089-e9f1-454c-a5fd-32712c7f4119","Type":"ContainerStarted","Data":"a8f9d1a0f43f94b53da8892599c834432580a548bdf7ce354336ed14886b632d"}
Nov 03 08:52:11.051653 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:11.051449    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-6a1442a9-69f5d84b5f-hxfs9"
Nov 03 08:52:11.168884 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:11.168821    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-6a1442a9-69f5d84b5f-hxfs9" podStartSLOduration=1.61609726 podStartE2EDuration="4.168803025s" totalImagesPullingTime="2.552705765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:52:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:52:11.167456092 +0000 UTC m=+755.436519527" watchObservedRunningTime="2026-11-03 08:52:11.168803025 +0000 UTC m=+755.437866456"
Nov 03 08:52:14.792579 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:14.792550    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 08:52:16.938442 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:52:16.938302    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:52:20.127077 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:20.127034    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-98d5dd-77466f785f-drltw"]
Nov 03 08:52:20.130616 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:20.130600    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-98d5dd-77466f785f-drltw"
Nov 03 08:52:20.189415 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:20.189386    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-98d5dd-77466f785f-drltw"]
Nov 03 08:52:20.237719 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:20.237688    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-98d5dd\""
Nov 03 08:52:20.237906 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:20.237740    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-98d5dd-dockercfg-k87fq\""
Nov 03 08:52:20.267839 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:20.267808    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4szmc\" (UniqueName: \"kubernetes.io/projected/576473d6-9fd5-4ce3-b85f-5053576c9de9-kube-api-access-4szmc\") pod \"payload-pre-processing-e2e-worker-w3-98d5dd-77466f785f-drltw\" (UID: \"576473d6-9fd5-4ce3-b85f-5053576c9de9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-98d5dd-77466f785f-drltw"
Nov 03 08:52:20.268017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:20.267954    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/576473d6-9fd5-4ce3-b85f-5053576c9de9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-98d5dd-77466f785f-drltw\" (UID: \"576473d6-9fd5-4ce3-b85f-5053576c9de9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-98d5dd-77466f785f-drltw"
Nov 03 08:52:20.289592 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:20.289566    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-98d5dd-d6c54bc97-xhlw4"]
Nov 03 08:52:20.293080 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:20.293064    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-98d5dd-d6c54bc97-xhlw4"
Nov 03 08:52:20.315718 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:20.315694    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-98d5dd-d6c54bc97-xhlw4"]
Nov 03 08:52:20.368722 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:20.368687    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4szmc\" (UniqueName: \"kubernetes.io/projected/576473d6-9fd5-4ce3-b85f-5053576c9de9-kube-api-access-4szmc\") pod \"payload-pre-processing-e2e-worker-w3-98d5dd-77466f785f-drltw\" (UID: \"576473d6-9fd5-4ce3-b85f-5053576c9de9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-98d5dd-77466f785f-drltw"
Nov 03 08:52:20.368878 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:20.368779    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/576473d6-9fd5-4ce3-b85f-5053576c9de9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-98d5dd-77466f785f-drltw\" (UID: \"576473d6-9fd5-4ce3-b85f-5053576c9de9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-98d5dd-77466f785f-drltw"
Nov 03 08:52:20.369428 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:20.369408    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/576473d6-9fd5-4ce3-b85f-5053576c9de9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-98d5dd-77466f785f-drltw\" (UID: \"576473d6-9fd5-4ce3-b85f-5053576c9de9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-98d5dd-77466f785f-drltw"
Nov 03 08:52:20.428606 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:20.428549    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4szmc\" (UniqueName: \"kubernetes.io/projected/576473d6-9fd5-4ce3-b85f-5053576c9de9-kube-api-access-4szmc\") pod \"payload-pre-processing-e2e-worker-w3-98d5dd-77466f785f-drltw\" (UID: \"576473d6-9fd5-4ce3-b85f-5053576c9de9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-98d5dd-77466f785f-drltw"
Nov 03 08:52:20.443275 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:20.443254    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-98d5dd-77466f785f-drltw"
Nov 03 08:52:20.469359 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:20.469334    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/600108c9-7889-4e9f-a213-dfe5e1be9ad6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-98d5dd-d6c54bc97-xhlw4\" (UID: \"600108c9-7889-4e9f-a213-dfe5e1be9ad6\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-98d5dd-d6c54bc97-xhlw4"
Nov 03 08:52:20.469460 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:20.469390    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnxxp\" (UniqueName: \"kubernetes.io/projected/600108c9-7889-4e9f-a213-dfe5e1be9ad6-kube-api-access-vnxxp\") pod \"payload-processing-e2e-worker-w3-98d5dd-d6c54bc97-xhlw4\" (UID: \"600108c9-7889-4e9f-a213-dfe5e1be9ad6\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-98d5dd-d6c54bc97-xhlw4"
Nov 03 08:52:20.570343 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:20.570316    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnxxp\" (UniqueName: \"kubernetes.io/projected/600108c9-7889-4e9f-a213-dfe5e1be9ad6-kube-api-access-vnxxp\") pod \"payload-processing-e2e-worker-w3-98d5dd-d6c54bc97-xhlw4\" (UID: \"600108c9-7889-4e9f-a213-dfe5e1be9ad6\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-98d5dd-d6c54bc97-xhlw4"
Nov 03 08:52:20.570515 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:20.570396    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/600108c9-7889-4e9f-a213-dfe5e1be9ad6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-98d5dd-d6c54bc97-xhlw4\" (UID: \"600108c9-7889-4e9f-a213-dfe5e1be9ad6\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-98d5dd-d6c54bc97-xhlw4"
Nov 03 08:52:20.571044 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:20.571026    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/600108c9-7889-4e9f-a213-dfe5e1be9ad6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-98d5dd-d6c54bc97-xhlw4\" (UID: \"600108c9-7889-4e9f-a213-dfe5e1be9ad6\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-98d5dd-d6c54bc97-xhlw4"
Nov 03 08:52:20.630787 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:20.630749    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnxxp\" (UniqueName: \"kubernetes.io/projected/600108c9-7889-4e9f-a213-dfe5e1be9ad6-kube-api-access-vnxxp\") pod \"payload-processing-e2e-worker-w3-98d5dd-d6c54bc97-xhlw4\" (UID: \"600108c9-7889-4e9f-a213-dfe5e1be9ad6\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-98d5dd-d6c54bc97-xhlw4"
Nov 03 08:52:20.814154 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:20.814130    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-98d5dd-77466f785f-drltw"]
Nov 03 08:52:20.817140 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:52:20.817112    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod576473d6_9fd5_4ce3_b85f_5053576c9de9.slice/crio-d8f64fa5504e5281a9403c03ebfd403097b922eb5509189934b767a1f7e56b43 WatchSource:0}: Error finding container d8f64fa5504e5281a9403c03ebfd403097b922eb5509189934b767a1f7e56b43: Status 404 returned error can't find the container with id d8f64fa5504e5281a9403c03ebfd403097b922eb5509189934b767a1f7e56b43
Nov 03 08:52:20.904997 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:20.904974    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-98d5dd-d6c54bc97-xhlw4"
Nov 03 08:52:21.101540 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:21.101444    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-98d5dd-77466f785f-drltw" event={"ID":"576473d6-9fd5-4ce3-b85f-5053576c9de9","Type":"ContainerStarted","Data":"39e5fc439e5888131c7e7717eb2dd7facb92fa9bb51188921ba5ec6f8f34a9f5"}
Nov 03 08:52:21.101540 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:21.101513    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-98d5dd-77466f785f-drltw"
Nov 03 08:52:21.101540 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:21.101540    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-98d5dd-77466f785f-drltw" event={"ID":"576473d6-9fd5-4ce3-b85f-5053576c9de9","Type":"ContainerStarted","Data":"d8f64fa5504e5281a9403c03ebfd403097b922eb5509189934b767a1f7e56b43"}
Nov 03 08:52:21.226119 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:21.226067    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-98d5dd-77466f785f-drltw" podStartSLOduration=1.226052176 podStartE2EDuration="1.226052176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:52:21.225076435 +0000 UTC m=+765.494139916" watchObservedRunningTime="2026-11-03 08:52:21.226052176 +0000 UTC m=+765.495115609"
Nov 03 08:52:21.282416 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:21.282397    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-98d5dd-d6c54bc97-xhlw4"]
Nov 03 08:52:21.284489 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:52:21.284466    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod600108c9_7889_4e9f_a213_dfe5e1be9ad6.slice/crio-f22bad2a9dcbaf280f2a580b8c5e732cabd5e1dbf379ad949b3bbe8b9c9f3d34 WatchSource:0}: Error finding container f22bad2a9dcbaf280f2a580b8c5e732cabd5e1dbf379ad949b3bbe8b9c9f3d34: Status 404 returned error can't find the container with id f22bad2a9dcbaf280f2a580b8c5e732cabd5e1dbf379ad949b3bbe8b9c9f3d34
Nov 03 08:52:22.058633 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:22.058601    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-6a1442a9-69f5d84b5f-hxfs9"
Nov 03 08:52:22.106834 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:22.106798    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-98d5dd-d6c54bc97-xhlw4" event={"ID":"600108c9-7889-4e9f-a213-dfe5e1be9ad6","Type":"ContainerStarted","Data":"39ba029e47ebe74f4193fb5b8082e9d7de3126ee6ee3bf03c6241658795a4658"}
Nov 03 08:52:22.107029 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:22.106862    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-98d5dd-d6c54bc97-xhlw4" event={"ID":"600108c9-7889-4e9f-a213-dfe5e1be9ad6","Type":"ContainerStarted","Data":"f22bad2a9dcbaf280f2a580b8c5e732cabd5e1dbf379ad949b3bbe8b9c9f3d34"}
Nov 03 08:52:22.107029 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:22.107015    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-98d5dd-d6c54bc97-xhlw4"
Nov 03 08:52:22.293221 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:22.293174    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-98d5dd-d6c54bc97-xhlw4" podStartSLOduration=2.293160262 podStartE2EDuration="2.293160262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:52:22.291339102 +0000 UTC m=+766.560402602" watchObservedRunningTime="2026-11-03 08:52:22.293160262 +0000 UTC m=+766.562223696"
Nov 03 08:52:22.856876 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:22.856846    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 08:52:32.110303 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:32.110275    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-98d5dd-77466f785f-drltw"
Nov 03 08:52:32.386789 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:32.386762    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 08:52:33.116000 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:33.115975    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-98d5dd-d6c54bc97-xhlw4"
Nov 03 08:52:43.304479 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:43.304448    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 08:52:46.936693 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:52:46.936653    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:52:47.518097 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:47.518065    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82eec5e9-jx54j"]
Nov 03 08:52:47.525451 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:47.525433    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82eec5e9-jx54j"
Nov 03 08:52:47.580513 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:47.580486    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82eec5e9-jx54j"]
Nov 03 08:52:47.624051 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:47.624020    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0da8e1d6-a1d7-4a1b-9281-dbde041b670e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-82eec5e9-jx54j\" (UID: \"0da8e1d6-a1d7-4a1b-9281-dbde041b670e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82eec5e9-jx54j"
Nov 03 08:52:47.624185 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:47.624056    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0da8e1d6-a1d7-4a1b-9281-dbde041b670e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-82eec5e9-jx54j\" (UID: \"0da8e1d6-a1d7-4a1b-9281-dbde041b670e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82eec5e9-jx54j"
Nov 03 08:52:47.624185 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:47.624076    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fjpq\" (UniqueName: \"kubernetes.io/projected/0da8e1d6-a1d7-4a1b-9281-dbde041b670e-kube-api-access-9fjpq\") pod \"maas-api-revoke-keys-e2e-ait-82eec5e9-jx54j\" (UID: \"0da8e1d6-a1d7-4a1b-9281-dbde041b670e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82eec5e9-jx54j"
Nov 03 08:52:47.628469 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:47.628454    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:52:47.628537 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:47.628488    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:52:47.628537 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:47.628496    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:52:47.628897 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:47.628883    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 08:52:47.725323 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:47.725296    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0da8e1d6-a1d7-4a1b-9281-dbde041b670e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-82eec5e9-jx54j\" (UID: \"0da8e1d6-a1d7-4a1b-9281-dbde041b670e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82eec5e9-jx54j"
Nov 03 08:52:47.725468 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:47.725332    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0da8e1d6-a1d7-4a1b-9281-dbde041b670e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-82eec5e9-jx54j\" (UID: \"0da8e1d6-a1d7-4a1b-9281-dbde041b670e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82eec5e9-jx54j"
Nov 03 08:52:47.725468 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:47.725353    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fjpq\" (UniqueName: \"kubernetes.io/projected/0da8e1d6-a1d7-4a1b-9281-dbde041b670e-kube-api-access-9fjpq\") pod \"maas-api-revoke-keys-e2e-ait-82eec5e9-jx54j\" (UID: \"0da8e1d6-a1d7-4a1b-9281-dbde041b670e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82eec5e9-jx54j"
Nov 03 08:52:47.726037 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:47.726021    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0da8e1d6-a1d7-4a1b-9281-dbde041b670e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-82eec5e9-jx54j\" (UID: \"0da8e1d6-a1d7-4a1b-9281-dbde041b670e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82eec5e9-jx54j"
Nov 03 08:52:47.727750 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:47.727733    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0da8e1d6-a1d7-4a1b-9281-dbde041b670e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-82eec5e9-jx54j\" (UID: \"0da8e1d6-a1d7-4a1b-9281-dbde041b670e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82eec5e9-jx54j"
Nov 03 08:52:47.785543 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:47.785484    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fjpq\" (UniqueName: \"kubernetes.io/projected/0da8e1d6-a1d7-4a1b-9281-dbde041b670e-kube-api-access-9fjpq\") pod \"maas-api-revoke-keys-e2e-ait-82eec5e9-jx54j\" (UID: \"0da8e1d6-a1d7-4a1b-9281-dbde041b670e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82eec5e9-jx54j"
Nov 03 08:52:47.837954 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:47.837927    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82eec5e9-jx54j"
Nov 03 08:52:48.210232 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:48.210205    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82eec5e9-jx54j"]
Nov 03 08:52:48.212510 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:52:48.212485    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da8e1d6_a1d7_4a1b_9281_dbde041b670e.slice/crio-5b91fe5f40d418a7967d403de25fa9e0ed2f972ca98e7687170f94d00f8f8880 WatchSource:0}: Error finding container 5b91fe5f40d418a7967d403de25fa9e0ed2f972ca98e7687170f94d00f8f8880: Status 404 returned error can't find the container with id 5b91fe5f40d418a7967d403de25fa9e0ed2f972ca98e7687170f94d00f8f8880
Nov 03 08:52:48.239037 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:48.239008    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82eec5e9-jx54j" event={"ID":"0da8e1d6-a1d7-4a1b-9281-dbde041b670e","Type":"ContainerStarted","Data":"5b91fe5f40d418a7967d403de25fa9e0ed2f972ca98e7687170f94d00f8f8880"}
Nov 03 08:52:48.519518 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:48.519440    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a1442a9-wtbhj"]
Nov 03 08:52:48.524416 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:48.524397    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a1442a9-wtbhj"
Nov 03 08:52:48.591234 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:48.591212    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a1442a9-wtbhj"]
Nov 03 08:52:48.634314 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:48.634285    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4r2w\" (UniqueName: \"kubernetes.io/projected/82be5f16-5aae-41c0-8c11-8a842fbf5ed5-kube-api-access-m4r2w\") pod \"maas-api-revoke-keys-e2e-ait-6a1442a9-wtbhj\" (UID: \"82be5f16-5aae-41c0-8c11-8a842fbf5ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a1442a9-wtbhj"
Nov 03 08:52:48.634448 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:48.634325    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82be5f16-5aae-41c0-8c11-8a842fbf5ed5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6a1442a9-wtbhj\" (UID: \"82be5f16-5aae-41c0-8c11-8a842fbf5ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a1442a9-wtbhj"
Nov 03 08:52:48.634448 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:48.634420    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82be5f16-5aae-41c0-8c11-8a842fbf5ed5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6a1442a9-wtbhj\" (UID: \"82be5f16-5aae-41c0-8c11-8a842fbf5ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a1442a9-wtbhj"
Nov 03 08:52:48.735927 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:48.735893    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4r2w\" (UniqueName: \"kubernetes.io/projected/82be5f16-5aae-41c0-8c11-8a842fbf5ed5-kube-api-access-m4r2w\") pod \"maas-api-revoke-keys-e2e-ait-6a1442a9-wtbhj\" (UID: \"82be5f16-5aae-41c0-8c11-8a842fbf5ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a1442a9-wtbhj"
Nov 03 08:52:48.736129 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:48.735963    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82be5f16-5aae-41c0-8c11-8a842fbf5ed5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6a1442a9-wtbhj\" (UID: \"82be5f16-5aae-41c0-8c11-8a842fbf5ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a1442a9-wtbhj"
Nov 03 08:52:48.736129 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:48.736008    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82be5f16-5aae-41c0-8c11-8a842fbf5ed5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6a1442a9-wtbhj\" (UID: \"82be5f16-5aae-41c0-8c11-8a842fbf5ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a1442a9-wtbhj"
Nov 03 08:52:48.736663 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:48.736643    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82be5f16-5aae-41c0-8c11-8a842fbf5ed5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6a1442a9-wtbhj\" (UID: \"82be5f16-5aae-41c0-8c11-8a842fbf5ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a1442a9-wtbhj"
Nov 03 08:52:48.738542 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:48.738520    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82be5f16-5aae-41c0-8c11-8a842fbf5ed5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6a1442a9-wtbhj\" (UID: \"82be5f16-5aae-41c0-8c11-8a842fbf5ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a1442a9-wtbhj"
Nov 03 08:52:48.795695 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:48.795632    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4r2w\" (UniqueName: \"kubernetes.io/projected/82be5f16-5aae-41c0-8c11-8a842fbf5ed5-kube-api-access-m4r2w\") pod \"maas-api-revoke-keys-e2e-ait-6a1442a9-wtbhj\" (UID: \"82be5f16-5aae-41c0-8c11-8a842fbf5ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a1442a9-wtbhj"
Nov 03 08:52:48.836782 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:48.836752    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a1442a9-wtbhj"
Nov 03 08:52:49.207759 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:49.207730    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a1442a9-wtbhj"]
Nov 03 08:52:49.209965 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:52:49.209920    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82be5f16_5aae_41c0_8c11_8a842fbf5ed5.slice/crio-a89eff28be6a2fc0accc66e1b2dfb2e1432f7e08faa6073db82ca82833afdff2 WatchSource:0}: Error finding container a89eff28be6a2fc0accc66e1b2dfb2e1432f7e08faa6073db82ca82833afdff2: Status 404 returned error can't find the container with id a89eff28be6a2fc0accc66e1b2dfb2e1432f7e08faa6073db82ca82833afdff2
Nov 03 08:52:49.245035 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:49.245010    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a1442a9-wtbhj" event={"ID":"82be5f16-5aae-41c0-8c11-8a842fbf5ed5","Type":"ContainerStarted","Data":"a89eff28be6a2fc0accc66e1b2dfb2e1432f7e08faa6073db82ca82833afdff2"}
Nov 03 08:52:52.261649 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:52.261620    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82eec5e9-jx54j" event={"ID":"0da8e1d6-a1d7-4a1b-9281-dbde041b670e","Type":"ContainerStarted","Data":"65ebd9aea50140772826a5642b72418603ee63eb9b85c77f72616c2eb8fd6e99"}
Nov 03 08:52:52.378617 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:52.378566    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82eec5e9-jx54j" podStartSLOduration=1.705611706 podStartE2EDuration="5.378548586s" totalImagesPullingTime="3.67293688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:52:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:52:52.377456551 +0000 UTC m=+796.646519986" watchObservedRunningTime="2026-11-03 08:52:52.378548586 +0000 UTC m=+796.647612022"
Nov 03 08:52:53.269015 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:53.268974    4473 generic.go:356] "Generic (PLEG): container finished" podID="0da8e1d6-a1d7-4a1b-9281-dbde041b670e" containerID="65ebd9aea50140772826a5642b72418603ee63eb9b85c77f72616c2eb8fd6e99" exitCode=0
Nov 03 08:52:53.269380 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:53.269041    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82eec5e9-jx54j" event={"ID":"0da8e1d6-a1d7-4a1b-9281-dbde041b670e","Type":"ContainerDied","Data":"65ebd9aea50140772826a5642b72418603ee63eb9b85c77f72616c2eb8fd6e99"}
Nov 03 08:52:53.270603 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:53.270582    4473 generic.go:356] "Generic (PLEG): container finished" podID="82be5f16-5aae-41c0-8c11-8a842fbf5ed5" containerID="dab4271f861945cd0862a2b9825165397fa64db79baf50f14978d7f23cf8343c" exitCode=0
Nov 03 08:52:53.270686 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:53.270666    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a1442a9-wtbhj" event={"ID":"82be5f16-5aae-41c0-8c11-8a842fbf5ed5","Type":"ContainerDied","Data":"dab4271f861945cd0862a2b9825165397fa64db79baf50f14978d7f23cf8343c"}
Nov 03 08:52:54.415046 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:54.415016    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a1442a9-wtbhj"
Nov 03 08:52:54.434673 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:54.434648    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82eec5e9-jx54j"
Nov 03 08:52:54.499724 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:54.499700    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82be5f16-5aae-41c0-8c11-8a842fbf5ed5-kube-api-access-m4r2w\" (UniqueName: \"kubernetes.io/projected/82be5f16-5aae-41c0-8c11-8a842fbf5ed5-kube-api-access-m4r2w\") pod \"82be5f16-5aae-41c0-8c11-8a842fbf5ed5\" (UID: \"82be5f16-5aae-41c0-8c11-8a842fbf5ed5\") "
Nov 03 08:52:54.499896 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:54.499741    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82be5f16-5aae-41c0-8c11-8a842fbf5ed5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82be5f16-5aae-41c0-8c11-8a842fbf5ed5-maas-api-openshift-ca\") pod \"82be5f16-5aae-41c0-8c11-8a842fbf5ed5\" (UID: \"82be5f16-5aae-41c0-8c11-8a842fbf5ed5\") "
Nov 03 08:52:54.499896 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:54.499845    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82be5f16-5aae-41c0-8c11-8a842fbf5ed5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82be5f16-5aae-41c0-8c11-8a842fbf5ed5-maas-api-xks-ca\") pod \"82be5f16-5aae-41c0-8c11-8a842fbf5ed5\" (UID: \"82be5f16-5aae-41c0-8c11-8a842fbf5ed5\") "
Nov 03 08:52:54.500171 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:54.500142    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82be5f16-5aae-41c0-8c11-8a842fbf5ed5-maas-api-openshift-ca" pod "82be5f16-5aae-41c0-8c11-8a842fbf5ed5" (UID: "82be5f16-5aae-41c0-8c11-8a842fbf5ed5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:52:54.502143 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:54.502121    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82be5f16-5aae-41c0-8c11-8a842fbf5ed5-maas-api-xks-ca" pod "82be5f16-5aae-41c0-8c11-8a842fbf5ed5" (UID: "82be5f16-5aae-41c0-8c11-8a842fbf5ed5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:52:54.502219 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:54.502192    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82be5f16-5aae-41c0-8c11-8a842fbf5ed5-kube-api-access-m4r2w" pod "82be5f16-5aae-41c0-8c11-8a842fbf5ed5" (UID: "82be5f16-5aae-41c0-8c11-8a842fbf5ed5"). InnerVolumeSpecName "kube-api-access-m4r2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:52:54.600626 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:54.600554    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da8e1d6-a1d7-4a1b-9281-dbde041b670e-kube-api-access-9fjpq\" (UniqueName: \"kubernetes.io/projected/0da8e1d6-a1d7-4a1b-9281-dbde041b670e-kube-api-access-9fjpq\") pod \"0da8e1d6-a1d7-4a1b-9281-dbde041b670e\" (UID: \"0da8e1d6-a1d7-4a1b-9281-dbde041b670e\") "
Nov 03 08:52:54.600626 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:54.600592    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0da8e1d6-a1d7-4a1b-9281-dbde041b670e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0da8e1d6-a1d7-4a1b-9281-dbde041b670e-maas-api-openshift-ca\") pod \"0da8e1d6-a1d7-4a1b-9281-dbde041b670e\" (UID: \"0da8e1d6-a1d7-4a1b-9281-dbde041b670e\") "
Nov 03 08:52:54.600814 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:54.600640    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0da8e1d6-a1d7-4a1b-9281-dbde041b670e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0da8e1d6-a1d7-4a1b-9281-dbde041b670e-maas-api-xks-ca\") pod \"0da8e1d6-a1d7-4a1b-9281-dbde041b670e\" (UID: \"0da8e1d6-a1d7-4a1b-9281-dbde041b670e\") "
Nov 03 08:52:54.601039 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:54.601014    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4r2w\" (UniqueName: \"kubernetes.io/projected/82be5f16-5aae-41c0-8c11-8a842fbf5ed5-kube-api-access-m4r2w\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:52:54.601119 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:54.601044    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82be5f16-5aae-41c0-8c11-8a842fbf5ed5-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:52:54.601119 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:54.601040    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da8e1d6-a1d7-4a1b-9281-dbde041b670e-maas-api-openshift-ca" pod "0da8e1d6-a1d7-4a1b-9281-dbde041b670e" (UID: "0da8e1d6-a1d7-4a1b-9281-dbde041b670e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:52:54.601119 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:54.601059    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82be5f16-5aae-41c0-8c11-8a842fbf5ed5-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:52:54.602801 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:54.602775    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da8e1d6-a1d7-4a1b-9281-dbde041b670e-maas-api-xks-ca" pod "0da8e1d6-a1d7-4a1b-9281-dbde041b670e" (UID: "0da8e1d6-a1d7-4a1b-9281-dbde041b670e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:52:54.602889 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:54.602815    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da8e1d6-a1d7-4a1b-9281-dbde041b670e-kube-api-access-9fjpq" pod "0da8e1d6-a1d7-4a1b-9281-dbde041b670e" (UID: "0da8e1d6-a1d7-4a1b-9281-dbde041b670e"). InnerVolumeSpecName "kube-api-access-9fjpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:52:54.702319 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:54.702293    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fjpq\" (UniqueName: \"kubernetes.io/projected/0da8e1d6-a1d7-4a1b-9281-dbde041b670e-kube-api-access-9fjpq\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:52:54.702319 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:54.702318    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0da8e1d6-a1d7-4a1b-9281-dbde041b670e-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:52:54.702494 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:54.702328    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0da8e1d6-a1d7-4a1b-9281-dbde041b670e-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:52:55.280871 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:55.280841    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82eec5e9-jx54j"
Nov 03 08:52:55.280871 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:55.280852    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82eec5e9-jx54j" event={"ID":"0da8e1d6-a1d7-4a1b-9281-dbde041b670e","Type":"ContainerDied","Data":"5b91fe5f40d418a7967d403de25fa9e0ed2f972ca98e7687170f94d00f8f8880"}
Nov 03 08:52:55.281129 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:55.280893    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b91fe5f40d418a7967d403de25fa9e0ed2f972ca98e7687170f94d00f8f8880"
Nov 03 08:52:55.282202 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:55.282181    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a1442a9-wtbhj"
Nov 03 08:52:55.282317 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:55.282205    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a1442a9-wtbhj" event={"ID":"82be5f16-5aae-41c0-8c11-8a842fbf5ed5","Type":"ContainerDied","Data":"a89eff28be6a2fc0accc66e1b2dfb2e1432f7e08faa6073db82ca82833afdff2"}
Nov 03 08:52:55.282317 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:55.282246    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a89eff28be6a2fc0accc66e1b2dfb2e1432f7e08faa6073db82ca82833afdff2"
Nov 03 08:52:59.090822 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:52:59.090796    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6a1442a9-69f5d84b5f-hxfs9"]
Nov 03 08:53:04.092054 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:04.092001    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-6a1442a9-69f5d84b5f-hxfs9" podUID="3a56b089-e9f1-454c-a5fd-32712c7f4119" containerName="payload-processing" containerID="cri-o://a8f9d1a0f43f94b53da8892599c834432580a548bdf7ce354336ed14886b632d" gracePeriod=25
Nov 03 08:53:04.327255 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:04.327221    4473 generic.go:356] "Generic (PLEG): container finished" podID="3a56b089-e9f1-454c-a5fd-32712c7f4119" containerID="a8f9d1a0f43f94b53da8892599c834432580a548bdf7ce354336ed14886b632d" exitCode=0
Nov 03 08:53:04.327393 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:04.327279    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6a1442a9-69f5d84b5f-hxfs9" event={"ID":"3a56b089-e9f1-454c-a5fd-32712c7f4119","Type":"ContainerDied","Data":"a8f9d1a0f43f94b53da8892599c834432580a548bdf7ce354336ed14886b632d"}
Nov 03 08:53:04.327393 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:04.327340    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6a1442a9-69f5d84b5f-hxfs9" event={"ID":"3a56b089-e9f1-454c-a5fd-32712c7f4119","Type":"ContainerDied","Data":"b3f02738ed9e50da5c2b4bd5b66119e258e57abe98bd4399d28f841be9c0c866"}
Nov 03 08:53:04.327393 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:04.327360    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3f02738ed9e50da5c2b4bd5b66119e258e57abe98bd4399d28f841be9c0c866"
Nov 03 08:53:04.340814 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:04.340797    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6a1442a9-69f5d84b5f-hxfs9"
Nov 03 08:53:04.488080 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:04.488052    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a56b089-e9f1-454c-a5fd-32712c7f4119-kube-api-access-jq686\" (UniqueName: \"kubernetes.io/projected/3a56b089-e9f1-454c-a5fd-32712c7f4119-kube-api-access-jq686\") pod \"3a56b089-e9f1-454c-a5fd-32712c7f4119\" (UID: \"3a56b089-e9f1-454c-a5fd-32712c7f4119\") "
Nov 03 08:53:04.488254 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:04.488191    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a56b089-e9f1-454c-a5fd-32712c7f4119-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a56b089-e9f1-454c-a5fd-32712c7f4119-plugins-config-volume\") pod \"3a56b089-e9f1-454c-a5fd-32712c7f4119\" (UID: \"3a56b089-e9f1-454c-a5fd-32712c7f4119\") "
Nov 03 08:53:04.488546 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:04.488523    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a56b089-e9f1-454c-a5fd-32712c7f4119-plugins-config-volume" pod "3a56b089-e9f1-454c-a5fd-32712c7f4119" (UID: "3a56b089-e9f1-454c-a5fd-32712c7f4119"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:53:04.490360 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:04.490335    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a56b089-e9f1-454c-a5fd-32712c7f4119-kube-api-access-jq686" pod "3a56b089-e9f1-454c-a5fd-32712c7f4119" (UID: "3a56b089-e9f1-454c-a5fd-32712c7f4119"). InnerVolumeSpecName "kube-api-access-jq686". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:53:04.589351 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:04.589313    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a56b089-e9f1-454c-a5fd-32712c7f4119-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:53:04.589351 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:04.589340    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq686\" (UniqueName: \"kubernetes.io/projected/3a56b089-e9f1-454c-a5fd-32712c7f4119-kube-api-access-jq686\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:53:04.862651 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:04.862572    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e658462-w5gzq"]
Nov 03 08:53:04.862991 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:04.862975    4473 cpu_manager.go:414] "containerMap: removing container" podUID="0da8e1d6-a1d7-4a1b-9281-dbde041b670e" containerName="revoke-keys"
Nov 03 08:53:04.863044 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:04.862994    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0da8e1d6-a1d7-4a1b-9281-dbde041b670e" containerName="revoke-keys"
Nov 03 08:53:04.863044 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:04.863028    4473 cpu_manager.go:414] "containerMap: removing container" podUID="3a56b089-e9f1-454c-a5fd-32712c7f4119" containerName="payload-processing"
Nov 03 08:53:04.863044 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:04.863033    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a56b089-e9f1-454c-a5fd-32712c7f4119" containerName="payload-processing"
Nov 03 08:53:04.863044 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:04.863040    4473 cpu_manager.go:414] "containerMap: removing container" podUID="82be5f16-5aae-41c0-8c11-8a842fbf5ed5" containerName="revoke-keys"
Nov 03 08:53:04.863159 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:04.863046    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82be5f16-5aae-41c0-8c11-8a842fbf5ed5" containerName="revoke-keys"
Nov 03 08:53:04.863159 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:04.863107    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="0da8e1d6-a1d7-4a1b-9281-dbde041b670e" containerName="revoke-keys"
Nov 03 08:53:04.863159 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:04.863121    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a56b089-e9f1-454c-a5fd-32712c7f4119" containerName="payload-processing"
Nov 03 08:53:04.863159 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:04.863127    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="82be5f16-5aae-41c0-8c11-8a842fbf5ed5" containerName="revoke-keys"
Nov 03 08:53:04.866266 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:04.866251    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e658462-w5gzq"
Nov 03 08:53:04.922625 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:04.922595    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:53:04.922896 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:04.922876    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 08:53:04.923012 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:04.922918    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:53:04.923012 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:04.922998    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:53:04.923871 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:04.923850    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e658462-w5gzq"]
Nov 03 08:53:04.992888 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:04.992855    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m5t5\" (UniqueName: \"kubernetes.io/projected/a9e5a66d-09c6-449c-b244-6508e55db876-kube-api-access-7m5t5\") pod \"maas-api-revoke-keys-e2e-derive-7e658462-w5gzq\" (UID: \"a9e5a66d-09c6-449c-b244-6508e55db876\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e658462-w5gzq"
Nov 03 08:53:04.993037 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:04.992903    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e5a66d-09c6-449c-b244-6508e55db876-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-7e658462-w5gzq\" (UID: \"a9e5a66d-09c6-449c-b244-6508e55db876\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e658462-w5gzq"
Nov 03 08:53:04.993086 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:04.993064    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e5a66d-09c6-449c-b244-6508e55db876-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-7e658462-w5gzq\" (UID: \"a9e5a66d-09c6-449c-b244-6508e55db876\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e658462-w5gzq"
Nov 03 08:53:05.094161 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:05.094130    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e5a66d-09c6-449c-b244-6508e55db876-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-7e658462-w5gzq\" (UID: \"a9e5a66d-09c6-449c-b244-6508e55db876\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e658462-w5gzq"
Nov 03 08:53:05.094528 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:05.094236    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m5t5\" (UniqueName: \"kubernetes.io/projected/a9e5a66d-09c6-449c-b244-6508e55db876-kube-api-access-7m5t5\") pod \"maas-api-revoke-keys-e2e-derive-7e658462-w5gzq\" (UID: \"a9e5a66d-09c6-449c-b244-6508e55db876\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e658462-w5gzq"
Nov 03 08:53:05.094528 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:05.094274    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e5a66d-09c6-449c-b244-6508e55db876-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-7e658462-w5gzq\" (UID: \"a9e5a66d-09c6-449c-b244-6508e55db876\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e658462-w5gzq"
Nov 03 08:53:05.094825 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:05.094794    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e5a66d-09c6-449c-b244-6508e55db876-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-7e658462-w5gzq\" (UID: \"a9e5a66d-09c6-449c-b244-6508e55db876\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e658462-w5gzq"
Nov 03 08:53:05.096770 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:05.096751    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e5a66d-09c6-449c-b244-6508e55db876-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-7e658462-w5gzq\" (UID: \"a9e5a66d-09c6-449c-b244-6508e55db876\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e658462-w5gzq"
Nov 03 08:53:05.155054 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:05.155033    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m5t5\" (UniqueName: \"kubernetes.io/projected/a9e5a66d-09c6-449c-b244-6508e55db876-kube-api-access-7m5t5\") pod \"maas-api-revoke-keys-e2e-derive-7e658462-w5gzq\" (UID: \"a9e5a66d-09c6-449c-b244-6508e55db876\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e658462-w5gzq"
Nov 03 08:53:05.178720 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:05.178695    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e658462-w5gzq"
Nov 03 08:53:05.331488 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:05.331452    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6a1442a9-69f5d84b5f-hxfs9"
Nov 03 08:53:05.503406 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:05.503337    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6a1442a9-69f5d84b5f-hxfs9"]
Nov 03 08:53:05.506056 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:05.506036    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6a1442a9-69f5d84b5f-hxfs9"]
Nov 03 08:53:05.555872 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:05.555849    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e658462-w5gzq"]
Nov 03 08:53:05.591325 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:53:05.591298    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e5a66d_09c6_449c_b244_6508e55db876.slice/crio-9c1931f18c07c766a139d73473b696d47b806d7df6ca03bbfb819ab33038012a WatchSource:0}: Error finding container 9c1931f18c07c766a139d73473b696d47b806d7df6ca03bbfb819ab33038012a: Status 404 returned error can't find the container with id 9c1931f18c07c766a139d73473b696d47b806d7df6ca03bbfb819ab33038012a
Nov 03 08:53:05.868557 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:05.868531    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a56b089-e9f1-454c-a5fd-32712c7f4119" path="/var/lib/kubelet/pods/3a56b089-e9f1-454c-a5fd-32712c7f4119/volumes"
Nov 03 08:53:05.979136 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:05.979098    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 08:53:06.337018 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:06.336980    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e658462-w5gzq" event={"ID":"a9e5a66d-09c6-449c-b244-6508e55db876","Type":"ContainerStarted","Data":"26a41d8cd2ec15af243332318d488ab2a19451be7fda333d74178eeb8ac12a60"}
Nov 03 08:53:06.337386 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:06.337059    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e658462-w5gzq" event={"ID":"a9e5a66d-09c6-449c-b244-6508e55db876","Type":"ContainerStarted","Data":"9c1931f18c07c766a139d73473b696d47b806d7df6ca03bbfb819ab33038012a"}
Nov 03 08:53:06.454437 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:06.454381    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e658462-w5gzq" podStartSLOduration=2.454366679 podStartE2EDuration="2.454366679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:53:06.452574818 +0000 UTC m=+810.721638274" watchObservedRunningTime="2026-11-03 08:53:06.454366679 +0000 UTC m=+810.723430113"
Nov 03 08:53:07.343261 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:07.343224    4473 generic.go:356] "Generic (PLEG): container finished" podID="a9e5a66d-09c6-449c-b244-6508e55db876" containerID="26a41d8cd2ec15af243332318d488ab2a19451be7fda333d74178eeb8ac12a60" exitCode=0
Nov 03 08:53:07.343760 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:07.343295    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e658462-w5gzq" event={"ID":"a9e5a66d-09c6-449c-b244-6508e55db876","Type":"ContainerDied","Data":"26a41d8cd2ec15af243332318d488ab2a19451be7fda333d74178eeb8ac12a60"}
Nov 03 08:53:08.490141 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:08.490119    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e658462-w5gzq"
Nov 03 08:53:08.540257 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:08.540225    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9e5a66d-09c6-449c-b244-6508e55db876-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e5a66d-09c6-449c-b244-6508e55db876-maas-api-openshift-ca\") pod \"a9e5a66d-09c6-449c-b244-6508e55db876\" (UID: \"a9e5a66d-09c6-449c-b244-6508e55db876\") "
Nov 03 08:53:08.540424 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:08.540341    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9e5a66d-09c6-449c-b244-6508e55db876-kube-api-access-7m5t5\" (UniqueName: \"kubernetes.io/projected/a9e5a66d-09c6-449c-b244-6508e55db876-kube-api-access-7m5t5\") pod \"a9e5a66d-09c6-449c-b244-6508e55db876\" (UID: \"a9e5a66d-09c6-449c-b244-6508e55db876\") "
Nov 03 08:53:08.540424 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:08.540386    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9e5a66d-09c6-449c-b244-6508e55db876-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e5a66d-09c6-449c-b244-6508e55db876-maas-api-xks-ca\") pod \"a9e5a66d-09c6-449c-b244-6508e55db876\" (UID: \"a9e5a66d-09c6-449c-b244-6508e55db876\") "
Nov 03 08:53:08.540598 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:08.540567    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9e5a66d-09c6-449c-b244-6508e55db876-maas-api-openshift-ca" pod "a9e5a66d-09c6-449c-b244-6508e55db876" (UID: "a9e5a66d-09c6-449c-b244-6508e55db876"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:53:08.542450 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:08.542421    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9e5a66d-09c6-449c-b244-6508e55db876-kube-api-access-7m5t5" pod "a9e5a66d-09c6-449c-b244-6508e55db876" (UID: "a9e5a66d-09c6-449c-b244-6508e55db876"). InnerVolumeSpecName "kube-api-access-7m5t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:53:08.542550 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:08.542465    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9e5a66d-09c6-449c-b244-6508e55db876-maas-api-xks-ca" pod "a9e5a66d-09c6-449c-b244-6508e55db876" (UID: "a9e5a66d-09c6-449c-b244-6508e55db876"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:53:08.641347 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:08.641320    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e5a66d-09c6-449c-b244-6508e55db876-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:53:08.641347 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:08.641346    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m5t5\" (UniqueName: \"kubernetes.io/projected/a9e5a66d-09c6-449c-b244-6508e55db876-kube-api-access-7m5t5\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:53:08.641518 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:08.641357    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e5a66d-09c6-449c-b244-6508e55db876-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:53:09.353362 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:09.353328    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e658462-w5gzq" event={"ID":"a9e5a66d-09c6-449c-b244-6508e55db876","Type":"ContainerDied","Data":"9c1931f18c07c766a139d73473b696d47b806d7df6ca03bbfb819ab33038012a"}
Nov 03 08:53:09.353362 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:09.353345    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e658462-w5gzq"
Nov 03 08:53:09.353567 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:09.353376    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c1931f18c07c766a139d73473b696d47b806d7df6ca03bbfb819ab33038012a"
Nov 03 08:53:09.658124 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:09.658092    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82eec5e9-jx54j"]
Nov 03 08:53:09.660722 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:09.660686    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82eec5e9-jx54j"]
Nov 03 08:53:09.868111 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:09.868086    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0da8e1d6-a1d7-4a1b-9281-dbde041b670e" path="/var/lib/kubelet/pods/0da8e1d6-a1d7-4a1b-9281-dbde041b670e/volumes"
Nov 03 08:53:10.732666 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:10.732633    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a1442a9-wtbhj"]
Nov 03 08:53:10.782850 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:10.782824    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a1442a9-wtbhj"]
Nov 03 08:53:11.868225 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:11.868199    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82be5f16-5aae-41c0-8c11-8a842fbf5ed5" path="/var/lib/kubelet/pods/82be5f16-5aae-41c0-8c11-8a842fbf5ed5/volumes"
Nov 03 08:53:13.655592 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:13.655558    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"]
Nov 03 08:53:13.656046 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:13.656032    4473 cpu_manager.go:414] "containerMap: removing container" podUID="a9e5a66d-09c6-449c-b244-6508e55db876" containerName="revoke-keys"
Nov 03 08:53:13.656100 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:13.656047    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9e5a66d-09c6-449c-b244-6508e55db876" containerName="revoke-keys"
Nov 03 08:53:13.656135 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:13.656112    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9e5a66d-09c6-449c-b244-6508e55db876" containerName="revoke-keys"
Nov 03 08:53:13.660348 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:13.660316    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"
Nov 03 08:53:13.687630 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:13.687600    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0cc45ff-b2a8-408b-9434-812476948826-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq\" (UID: \"a0cc45ff-b2a8-408b-9434-812476948826\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"
Nov 03 08:53:13.687630 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:13.687635    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0cc45ff-b2a8-408b-9434-812476948826-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq\" (UID: \"a0cc45ff-b2a8-408b-9434-812476948826\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"
Nov 03 08:53:13.687834 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:13.687653    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0cc45ff-b2a8-408b-9434-812476948826-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq\" (UID: \"a0cc45ff-b2a8-408b-9434-812476948826\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"
Nov 03 08:53:13.687834 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:13.687793    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0cc45ff-b2a8-408b-9434-812476948826-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq\" (UID: \"a0cc45ff-b2a8-408b-9434-812476948826\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"
Nov 03 08:53:13.687956 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:13.687841    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0cc45ff-b2a8-408b-9434-812476948826-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq\" (UID: \"a0cc45ff-b2a8-408b-9434-812476948826\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"
Nov 03 08:53:13.687956 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:13.687874    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0cc45ff-b2a8-408b-9434-812476948826-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq\" (UID: \"a0cc45ff-b2a8-408b-9434-812476948826\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"
Nov 03 08:53:13.687956 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:13.687892    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0cc45ff-b2a8-408b-9434-812476948826-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq\" (UID: \"a0cc45ff-b2a8-408b-9434-812476948826\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"
Nov 03 08:53:13.688112 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:13.688005    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0cc45ff-b2a8-408b-9434-812476948826-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq\" (UID: \"a0cc45ff-b2a8-408b-9434-812476948826\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"
Nov 03 08:53:13.688112 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:13.688026    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5q2f\" (UniqueName: \"kubernetes.io/projected/a0cc45ff-b2a8-408b-9434-812476948826-kube-api-access-r5q2f\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq\" (UID: \"a0cc45ff-b2a8-408b-9434-812476948826\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"
Nov 03 08:53:13.738377 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:13.738346    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"]
Nov 03 08:53:13.768746 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:13.768700    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-77207e-openshift-default-dockercfg-rcrfh\""
Nov 03 08:53:13.789205 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:13.789175    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0cc45ff-b2a8-408b-9434-812476948826-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq\" (UID: \"a0cc45ff-b2a8-408b-9434-812476948826\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"
Nov 03 08:53:13.789364 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:13.789210    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0cc45ff-b2a8-408b-9434-812476948826-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq\" (UID: \"a0cc45ff-b2a8-408b-9434-812476948826\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"
Nov 03 08:53:13.789364 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:13.789242    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0cc45ff-b2a8-408b-9434-812476948826-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq\" (UID: \"a0cc45ff-b2a8-408b-9434-812476948826\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"
Nov 03 08:53:13.789364 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:13.789264    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0cc45ff-b2a8-408b-9434-812476948826-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq\" (UID: \"a0cc45ff-b2a8-408b-9434-812476948826\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"
Nov 03 08:53:13.789364 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:13.789326    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0cc45ff-b2a8-408b-9434-812476948826-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq\" (UID: \"a0cc45ff-b2a8-408b-9434-812476948826\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"
Nov 03 08:53:13.789566 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:13.789461    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5q2f\" (UniqueName: \"kubernetes.io/projected/a0cc45ff-b2a8-408b-9434-812476948826-kube-api-access-r5q2f\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq\" (UID: \"a0cc45ff-b2a8-408b-9434-812476948826\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"
Nov 03 08:53:13.789566 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:13.789525    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0cc45ff-b2a8-408b-9434-812476948826-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq\" (UID: \"a0cc45ff-b2a8-408b-9434-812476948826\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"
Nov 03 08:53:13.789566 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:13.789557    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0cc45ff-b2a8-408b-9434-812476948826-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq\" (UID: \"a0cc45ff-b2a8-408b-9434-812476948826\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"
Nov 03 08:53:13.789711 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:13.789583    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0cc45ff-b2a8-408b-9434-812476948826-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq\" (UID: \"a0cc45ff-b2a8-408b-9434-812476948826\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"
Nov 03 08:53:13.789711 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:13.789608    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0cc45ff-b2a8-408b-9434-812476948826-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq\" (UID: \"a0cc45ff-b2a8-408b-9434-812476948826\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"
Nov 03 08:53:13.789711 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:13.789678    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0cc45ff-b2a8-408b-9434-812476948826-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq\" (UID: \"a0cc45ff-b2a8-408b-9434-812476948826\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"
Nov 03 08:53:13.789711 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:13.789696    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0cc45ff-b2a8-408b-9434-812476948826-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq\" (UID: \"a0cc45ff-b2a8-408b-9434-812476948826\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"
Nov 03 08:53:13.789989 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:13.789965    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0cc45ff-b2a8-408b-9434-812476948826-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq\" (UID: \"a0cc45ff-b2a8-408b-9434-812476948826\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"
Nov 03 08:53:13.790147 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:13.789989    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0cc45ff-b2a8-408b-9434-812476948826-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq\" (UID: \"a0cc45ff-b2a8-408b-9434-812476948826\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"
Nov 03 08:53:13.791711 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:13.791689    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0cc45ff-b2a8-408b-9434-812476948826-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq\" (UID: \"a0cc45ff-b2a8-408b-9434-812476948826\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"
Nov 03 08:53:13.792012 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:13.791995    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0cc45ff-b2a8-408b-9434-812476948826-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq\" (UID: \"a0cc45ff-b2a8-408b-9434-812476948826\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"
Nov 03 08:53:13.848745 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:13.848702    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0cc45ff-b2a8-408b-9434-812476948826-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq\" (UID: \"a0cc45ff-b2a8-408b-9434-812476948826\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"
Nov 03 08:53:13.848896 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:13.848836    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5q2f\" (UniqueName: \"kubernetes.io/projected/a0cc45ff-b2a8-408b-9434-812476948826-kube-api-access-r5q2f\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq\" (UID: \"a0cc45ff-b2a8-408b-9434-812476948826\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"
Nov 03 08:53:13.972735 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:13.972646    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"
Nov 03 08:53:14.412471 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:14.412438    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"]
Nov 03 08:53:14.419603 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:53:14.419573    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0cc45ff_b2a8_408b_9434_812476948826.slice/crio-c07c54532b30baf4ced3c70861620f3dd25adf262552be0a8f71af7b3c736abc WatchSource:0}: Error finding container c07c54532b30baf4ced3c70861620f3dd25adf262552be0a8f71af7b3c736abc: Status 404 returned error can't find the container with id c07c54532b30baf4ced3c70861620f3dd25adf262552be0a8f71af7b3c736abc
Nov 03 08:53:14.421921 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:14.421889    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 08:53:14.422009 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:14.421975    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 08:53:14.422063 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:14.422016    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 08:53:15.388248 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:15.388216    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq" event={"ID":"a0cc45ff-b2a8-408b-9434-812476948826","Type":"ContainerStarted","Data":"b2f3d4be138ef4d26a1147dd59695a0aa4350e0b7757865108d05a43d6b351f6"}
Nov 03 08:53:15.388638 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:15.388264    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq" event={"ID":"a0cc45ff-b2a8-408b-9434-812476948826","Type":"ContainerStarted","Data":"c07c54532b30baf4ced3c70861620f3dd25adf262552be0a8f71af7b3c736abc"}
Nov 03 08:53:15.560690 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:15.560631    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq" podStartSLOduration=2.560610962 podStartE2EDuration="2.560610962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:53:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:53:15.557193563 +0000 UTC m=+819.826257033" watchObservedRunningTime="2026-11-03 08:53:15.560610962 +0000 UTC m=+819.829674400"
Nov 03 08:53:15.973416 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:15.973383    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"
Nov 03 08:53:15.978905 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:15.978874    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"
Nov 03 08:53:16.393430 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:16.393401    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"
Nov 03 08:53:16.394318 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:16.394302    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"
Nov 03 08:53:16.936690 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:53:16.936655    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:53:21.716183 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:21.716152    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e658462-w5gzq"]
Nov 03 08:53:21.774447 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:21.774411    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7e658462-w5gzq"]
Nov 03 08:53:21.869456 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:21.869424    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9e5a66d-09c6-449c-b244-6508e55db876" path="/var/lib/kubelet/pods/a9e5a66d-09c6-449c-b244-6508e55db876/volumes"
Nov 03 08:53:21.879764 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:21.879739    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 08:53:36.911575 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:36.911536    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-910d50-7db8cfbf6c-w8z4h"]
Nov 03 08:53:36.919078 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:36.919054    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-910d50-7db8cfbf6c-w8z4h"
Nov 03 08:53:36.972516 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:36.972489    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-910d50-7db8cfbf6c-w8z4h"]
Nov 03 08:53:37.021160 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:37.021126    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-910d50\""
Nov 03 08:53:37.021355 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:37.021192    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-910d50-dockercfg-484nn\""
Nov 03 08:53:37.037186 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:37.037156    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/960d4e1b-7409-4a75-83f4-26289cb2b694-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-910d50-7db8cfbf6c-w8z4h\" (UID: \"960d4e1b-7409-4a75-83f4-26289cb2b694\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-910d50-7db8cfbf6c-w8z4h"
Nov 03 08:53:37.037495 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:37.037286    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv7zl\" (UniqueName: \"kubernetes.io/projected/960d4e1b-7409-4a75-83f4-26289cb2b694-kube-api-access-hv7zl\") pod \"payload-pre-processing-e2e-worker-w0-910d50-7db8cfbf6c-w8z4h\" (UID: \"960d4e1b-7409-4a75-83f4-26289cb2b694\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-910d50-7db8cfbf6c-w8z4h"
Nov 03 08:53:37.138793 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:37.138754    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv7zl\" (UniqueName: \"kubernetes.io/projected/960d4e1b-7409-4a75-83f4-26289cb2b694-kube-api-access-hv7zl\") pod \"payload-pre-processing-e2e-worker-w0-910d50-7db8cfbf6c-w8z4h\" (UID: \"960d4e1b-7409-4a75-83f4-26289cb2b694\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-910d50-7db8cfbf6c-w8z4h"
Nov 03 08:53:37.138983 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:37.138862    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/960d4e1b-7409-4a75-83f4-26289cb2b694-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-910d50-7db8cfbf6c-w8z4h\" (UID: \"960d4e1b-7409-4a75-83f4-26289cb2b694\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-910d50-7db8cfbf6c-w8z4h"
Nov 03 08:53:37.139541 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:37.139518    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/960d4e1b-7409-4a75-83f4-26289cb2b694-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-910d50-7db8cfbf6c-w8z4h\" (UID: \"960d4e1b-7409-4a75-83f4-26289cb2b694\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-910d50-7db8cfbf6c-w8z4h"
Nov 03 08:53:37.198383 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:37.198323    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv7zl\" (UniqueName: \"kubernetes.io/projected/960d4e1b-7409-4a75-83f4-26289cb2b694-kube-api-access-hv7zl\") pod \"payload-pre-processing-e2e-worker-w0-910d50-7db8cfbf6c-w8z4h\" (UID: \"960d4e1b-7409-4a75-83f4-26289cb2b694\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-910d50-7db8cfbf6c-w8z4h"
Nov 03 08:53:37.231870 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:37.231844    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-910d50-7db8cfbf6c-w8z4h"
Nov 03 08:53:37.605909 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:37.605885    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-910d50-7db8cfbf6c-w8z4h"]
Nov 03 08:53:37.607324 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:53:37.607298    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod960d4e1b_7409_4a75_83f4_26289cb2b694.slice/crio-77e37edf41981fdaba3f753cbae7beef9a2e3fd4c727f062b0c9c69307928b8f WatchSource:0}: Error finding container 77e37edf41981fdaba3f753cbae7beef9a2e3fd4c727f062b0c9c69307928b8f: Status 404 returned error can't find the container with id 77e37edf41981fdaba3f753cbae7beef9a2e3fd4c727f062b0c9c69307928b8f
Nov 03 08:53:38.520846 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:38.520816    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-910d50-7db8cfbf6c-w8z4h" event={"ID":"960d4e1b-7409-4a75-83f4-26289cb2b694","Type":"ContainerStarted","Data":"14e14f7a4445a22d1a082137713ea9d092a41b39400dbf88f167bdfd57c632a1"}
Nov 03 08:53:38.521268 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:38.520866    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-910d50-7db8cfbf6c-w8z4h" event={"ID":"960d4e1b-7409-4a75-83f4-26289cb2b694","Type":"ContainerStarted","Data":"77e37edf41981fdaba3f753cbae7beef9a2e3fd4c727f062b0c9c69307928b8f"}
Nov 03 08:53:38.521268 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:38.520930    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-910d50-7db8cfbf6c-w8z4h"
Nov 03 08:53:38.637811 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:38.637751    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-910d50-7db8cfbf6c-w8z4h" podStartSLOduration=2.637736331 podStartE2EDuration="2.637736331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:53:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:53:38.635986074 +0000 UTC m=+842.905049580" watchObservedRunningTime="2026-11-03 08:53:38.637736331 +0000 UTC m=+842.906799765"
Nov 03 08:53:40.992449 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:40.992358    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 08:53:46.937303 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:53:46.937206    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:53:48.384888 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:48.380117    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6"]
Nov 03 08:53:48.386154 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:48.386134    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6"
Nov 03 08:53:48.490037 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:48.490004    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-98d5dd\"/\"kube-root-ca.crt\""
Nov 03 08:53:48.490037 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:48.490004    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-98d5dd\"/\"premium-w3-98d5dd-kserve-self-signed-certs\""
Nov 03 08:53:48.490037 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:48.490018    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-98d5dd\"/\"default-dockercfg-dftgt\""
Nov 03 08:53:48.490584 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:48.490550    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-98d5dd\"/\"openshift-service-ca.crt\""
Nov 03 08:53:48.494054 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:48.494029    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6"]
Nov 03 08:53:48.567622 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:48.567583    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25c366b3-b919-462b-b69a-82ce9dbd7436-tmp-dir\") pod \"premium-w3-98d5dd-kserve-c98b77cd9-xq2d6\" (UID: \"25c366b3-b919-462b-b69a-82ce9dbd7436\") " pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6"
Nov 03 08:53:48.567622 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:48.567628    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25c366b3-b919-462b-b69a-82ce9dbd7436-tls-certs\") pod \"premium-w3-98d5dd-kserve-c98b77cd9-xq2d6\" (UID: \"25c366b3-b919-462b-b69a-82ce9dbd7436\") " pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6"
Nov 03 08:53:48.567899 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:48.567652    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25c366b3-b919-462b-b69a-82ce9dbd7436-home\") pod \"premium-w3-98d5dd-kserve-c98b77cd9-xq2d6\" (UID: \"25c366b3-b919-462b-b69a-82ce9dbd7436\") " pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6"
Nov 03 08:53:48.567899 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:48.567795    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25c366b3-b919-462b-b69a-82ce9dbd7436-model-cache\") pod \"premium-w3-98d5dd-kserve-c98b77cd9-xq2d6\" (UID: \"25c366b3-b919-462b-b69a-82ce9dbd7436\") " pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6"
Nov 03 08:53:48.567899 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:48.567821    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv8lx\" (UniqueName: \"kubernetes.io/projected/25c366b3-b919-462b-b69a-82ce9dbd7436-kube-api-access-mv8lx\") pod \"premium-w3-98d5dd-kserve-c98b77cd9-xq2d6\" (UID: \"25c366b3-b919-462b-b69a-82ce9dbd7436\") " pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6"
Nov 03 08:53:48.567899 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:48.567839    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25c366b3-b919-462b-b69a-82ce9dbd7436-dshm\") pod \"premium-w3-98d5dd-kserve-c98b77cd9-xq2d6\" (UID: \"25c366b3-b919-462b-b69a-82ce9dbd7436\") " pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6"
Nov 03 08:53:48.669403 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:48.669311    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25c366b3-b919-462b-b69a-82ce9dbd7436-tmp-dir\") pod \"premium-w3-98d5dd-kserve-c98b77cd9-xq2d6\" (UID: \"25c366b3-b919-462b-b69a-82ce9dbd7436\") " pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6"
Nov 03 08:53:48.669584 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:48.669400    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25c366b3-b919-462b-b69a-82ce9dbd7436-tls-certs\") pod \"premium-w3-98d5dd-kserve-c98b77cd9-xq2d6\" (UID: \"25c366b3-b919-462b-b69a-82ce9dbd7436\") " pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6"
Nov 03 08:53:48.669584 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:48.669448    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25c366b3-b919-462b-b69a-82ce9dbd7436-home\") pod \"premium-w3-98d5dd-kserve-c98b77cd9-xq2d6\" (UID: \"25c366b3-b919-462b-b69a-82ce9dbd7436\") " pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6"
Nov 03 08:53:48.669584 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:48.669526    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25c366b3-b919-462b-b69a-82ce9dbd7436-model-cache\") pod \"premium-w3-98d5dd-kserve-c98b77cd9-xq2d6\" (UID: \"25c366b3-b919-462b-b69a-82ce9dbd7436\") " pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6"
Nov 03 08:53:48.669584 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:48.669550    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv8lx\" (UniqueName: \"kubernetes.io/projected/25c366b3-b919-462b-b69a-82ce9dbd7436-kube-api-access-mv8lx\") pod \"premium-w3-98d5dd-kserve-c98b77cd9-xq2d6\" (UID: \"25c366b3-b919-462b-b69a-82ce9dbd7436\") " pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6"
Nov 03 08:53:48.669584 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:48.669578    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25c366b3-b919-462b-b69a-82ce9dbd7436-dshm\") pod \"premium-w3-98d5dd-kserve-c98b77cd9-xq2d6\" (UID: \"25c366b3-b919-462b-b69a-82ce9dbd7436\") " pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6"
Nov 03 08:53:48.669870 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:48.669763    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25c366b3-b919-462b-b69a-82ce9dbd7436-tmp-dir\") pod \"premium-w3-98d5dd-kserve-c98b77cd9-xq2d6\" (UID: \"25c366b3-b919-462b-b69a-82ce9dbd7436\") " pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6"
Nov 03 08:53:48.670058 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:48.670034    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25c366b3-b919-462b-b69a-82ce9dbd7436-model-cache\") pod \"premium-w3-98d5dd-kserve-c98b77cd9-xq2d6\" (UID: \"25c366b3-b919-462b-b69a-82ce9dbd7436\") " pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6"
Nov 03 08:53:48.670430 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:48.670401    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25c366b3-b919-462b-b69a-82ce9dbd7436-home\") pod \"premium-w3-98d5dd-kserve-c98b77cd9-xq2d6\" (UID: \"25c366b3-b919-462b-b69a-82ce9dbd7436\") " pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6"
Nov 03 08:53:48.672336 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:48.672306    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25c366b3-b919-462b-b69a-82ce9dbd7436-dshm\") pod \"premium-w3-98d5dd-kserve-c98b77cd9-xq2d6\" (UID: \"25c366b3-b919-462b-b69a-82ce9dbd7436\") " pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6"
Nov 03 08:53:48.672449 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:48.672316    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25c366b3-b919-462b-b69a-82ce9dbd7436-tls-certs\") pod \"premium-w3-98d5dd-kserve-c98b77cd9-xq2d6\" (UID: \"25c366b3-b919-462b-b69a-82ce9dbd7436\") " pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6"
Nov 03 08:53:48.730575 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:48.730532    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv8lx\" (UniqueName: \"kubernetes.io/projected/25c366b3-b919-462b-b69a-82ce9dbd7436-kube-api-access-mv8lx\") pod \"premium-w3-98d5dd-kserve-c98b77cd9-xq2d6\" (UID: \"25c366b3-b919-462b-b69a-82ce9dbd7436\") " pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6"
Nov 03 08:53:49.000561 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:49.000488    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6"
Nov 03 08:53:49.376030 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:49.375994    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6"]
Nov 03 08:53:49.379510 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:53:49.379485    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25c366b3_b919_462b_b69a_82ce9dbd7436.slice/crio-a21883144068b3e789bfa33f97081118438224c855a18ff096001606835d1e9b WatchSource:0}: Error finding container a21883144068b3e789bfa33f97081118438224c855a18ff096001606835d1e9b: Status 404 returned error can't find the container with id a21883144068b3e789bfa33f97081118438224c855a18ff096001606835d1e9b
Nov 03 08:53:49.530085 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:49.530052    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-910d50-7db8cfbf6c-w8z4h"
Nov 03 08:53:49.583469 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:49.583436    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6" event={"ID":"25c366b3-b919-462b-b69a-82ce9dbd7436","Type":"ContainerStarted","Data":"a21883144068b3e789bfa33f97081118438224c855a18ff096001606835d1e9b"}
Nov 03 08:53:50.591194 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:50.591160    4473 generic.go:356] "Generic (PLEG): container finished" podID="a0cc45ff-b2a8-408b-9434-812476948826" containerID="b2f3d4be138ef4d26a1147dd59695a0aa4350e0b7757865108d05a43d6b351f6" exitCode=0
Nov 03 08:53:50.591650 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:50.591228    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq" event={"ID":"a0cc45ff-b2a8-408b-9434-812476948826","Type":"ContainerDied","Data":"b2f3d4be138ef4d26a1147dd59695a0aa4350e0b7757865108d05a43d6b351f6"}
Nov 03 08:53:50.591650 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:50.591589    4473 scope.go:122] "RemoveContainer" containerID="b2f3d4be138ef4d26a1147dd59695a0aa4350e0b7757865108d05a43d6b351f6"
Nov 03 08:53:50.592972 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:50.592913    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 08:53:50.593098 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:50.593004    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 08:53:50.593098 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:50.593045    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 08:53:51.598173 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:51.598136    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq" event={"ID":"a0cc45ff-b2a8-408b-9434-812476948826","Type":"ContainerStarted","Data":"3d30a47e1d3a192db76a92a4b3f530ddfa0f04045eeca9995de5e75ad85c0945"}
Nov 03 08:53:51.599737 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:51.599711    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6" event={"ID":"25c366b3-b919-462b-b69a-82ce9dbd7436","Type":"ContainerStarted","Data":"b59b9360a0b2d91ca4d5785943ab0bfe3c1900086465341d125d6f741bca7861"}
Nov 03 08:53:51.885836 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:51.885779    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6" podStartSLOduration=2.363760536 podStartE2EDuration="3.885761172s" totalImagesPullingTime="1.522000636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:53:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:53:51.88450194 +0000 UTC m=+856.153565398" watchObservedRunningTime="2026-11-03 08:53:51.885761172 +0000 UTC m=+856.154824606"
Nov 03 08:53:51.973822 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:51.973788    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"
Nov 03 08:53:51.974031 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:51.974008    4473 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 03 08:53:51.974086 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:51.974058    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq" podUID="a0cc45ff-b2a8-408b-9434-812476948826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 03 08:53:52.003515 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:52.003481    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6"
Nov 03 08:53:52.476320 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:52.476289    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 08:53:52.973582 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:52.973547    4473 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 03 08:53:52.974056 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:52.973600    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq" podUID="a0cc45ff-b2a8-408b-9434-812476948826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 03 08:53:53.973146 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:53.973115    4473 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 03 08:53:53.973342 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:53.973171    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq" podUID="a0cc45ff-b2a8-408b-9434-812476948826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 03 08:53:54.973675 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:54.973639    4473 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 03 08:53:54.974100 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:54.973706    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq" podUID="a0cc45ff-b2a8-408b-9434-812476948826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 03 08:53:55.973739 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:55.973704    4473 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 03 08:53:55.974183 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:55.973767    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq" podUID="a0cc45ff-b2a8-408b-9434-812476948826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 03 08:53:56.976879 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:56.976852    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"
Nov 03 08:53:56.977286 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:56.977227    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"
Nov 03 08:53:56.977925 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:56.977905    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"
Nov 03 08:53:59.000848 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:59.000818    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6"
Nov 03 08:53:59.012967 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:59.012924    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6"
Nov 03 08:53:59.668093 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:53:59.668064    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6"
Nov 03 08:54:06.392298 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:06.392265    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 08:54:16.944704 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:54:16.944642    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:54:19.753573 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:19.753533    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh"]
Nov 03 08:54:19.758566 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:19.758546    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh"
Nov 03 08:54:19.815720 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:19.815688    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh"]
Nov 03 08:54:19.863715 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:19.863690    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-18f9da\"/\"kube-root-ca.crt\""
Nov 03 08:54:19.863878 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:19.863716    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-18f9da\"/\"openshift-service-ca.crt\""
Nov 03 08:54:19.864104 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:19.864086    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-18f9da\"/\"default-dockercfg-wmwxj\""
Nov 03 08:54:19.864190 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:19.864123    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-18f9da\"/\"simulator-w4-18f9da-kserve-self-signed-certs\""
Nov 03 08:54:19.912116 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:19.912065    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00bab831-c667-415d-b6b0-81fff4d57762-home\") pod \"simulator-w4-18f9da-kserve-66dd7c494b-x4zlh\" (UID: \"00bab831-c667-415d-b6b0-81fff4d57762\") " pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh"
Nov 03 08:54:19.912116 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:19.912114    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqjlv\" (UniqueName: \"kubernetes.io/projected/00bab831-c667-415d-b6b0-81fff4d57762-kube-api-access-bqjlv\") pod \"simulator-w4-18f9da-kserve-66dd7c494b-x4zlh\" (UID: \"00bab831-c667-415d-b6b0-81fff4d57762\") " pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh"
Nov 03 08:54:19.912362 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:19.912271    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00bab831-c667-415d-b6b0-81fff4d57762-tmp-dir\") pod \"simulator-w4-18f9da-kserve-66dd7c494b-x4zlh\" (UID: \"00bab831-c667-415d-b6b0-81fff4d57762\") " pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh"
Nov 03 08:54:19.912362 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:19.912306    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00bab831-c667-415d-b6b0-81fff4d57762-dshm\") pod \"simulator-w4-18f9da-kserve-66dd7c494b-x4zlh\" (UID: \"00bab831-c667-415d-b6b0-81fff4d57762\") " pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh"
Nov 03 08:54:19.912433 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:19.912359    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00bab831-c667-415d-b6b0-81fff4d57762-model-cache\") pod \"simulator-w4-18f9da-kserve-66dd7c494b-x4zlh\" (UID: \"00bab831-c667-415d-b6b0-81fff4d57762\") " pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh"
Nov 03 08:54:19.912433 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:19.912411    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00bab831-c667-415d-b6b0-81fff4d57762-tls-certs\") pod \"simulator-w4-18f9da-kserve-66dd7c494b-x4zlh\" (UID: \"00bab831-c667-415d-b6b0-81fff4d57762\") " pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh"
Nov 03 08:54:20.016372 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:20.016281    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00bab831-c667-415d-b6b0-81fff4d57762-tmp-dir\") pod \"simulator-w4-18f9da-kserve-66dd7c494b-x4zlh\" (UID: \"00bab831-c667-415d-b6b0-81fff4d57762\") " pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh"
Nov 03 08:54:20.016372 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:20.016333    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00bab831-c667-415d-b6b0-81fff4d57762-dshm\") pod \"simulator-w4-18f9da-kserve-66dd7c494b-x4zlh\" (UID: \"00bab831-c667-415d-b6b0-81fff4d57762\") " pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh"
Nov 03 08:54:20.016569 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:20.016386    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00bab831-c667-415d-b6b0-81fff4d57762-model-cache\") pod \"simulator-w4-18f9da-kserve-66dd7c494b-x4zlh\" (UID: \"00bab831-c667-415d-b6b0-81fff4d57762\") " pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh"
Nov 03 08:54:20.016569 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:20.016434    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00bab831-c667-415d-b6b0-81fff4d57762-tls-certs\") pod \"simulator-w4-18f9da-kserve-66dd7c494b-x4zlh\" (UID: \"00bab831-c667-415d-b6b0-81fff4d57762\") " pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh"
Nov 03 08:54:20.016569 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:20.016509    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00bab831-c667-415d-b6b0-81fff4d57762-home\") pod \"simulator-w4-18f9da-kserve-66dd7c494b-x4zlh\" (UID: \"00bab831-c667-415d-b6b0-81fff4d57762\") " pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh"
Nov 03 08:54:20.016569 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:20.016538    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqjlv\" (UniqueName: \"kubernetes.io/projected/00bab831-c667-415d-b6b0-81fff4d57762-kube-api-access-bqjlv\") pod \"simulator-w4-18f9da-kserve-66dd7c494b-x4zlh\" (UID: \"00bab831-c667-415d-b6b0-81fff4d57762\") " pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh"
Nov 03 08:54:20.017526 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:20.017469    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00bab831-c667-415d-b6b0-81fff4d57762-model-cache\") pod \"simulator-w4-18f9da-kserve-66dd7c494b-x4zlh\" (UID: \"00bab831-c667-415d-b6b0-81fff4d57762\") " pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh"
Nov 03 08:54:20.017526 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:20.017499    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00bab831-c667-415d-b6b0-81fff4d57762-home\") pod \"simulator-w4-18f9da-kserve-66dd7c494b-x4zlh\" (UID: \"00bab831-c667-415d-b6b0-81fff4d57762\") " pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh"
Nov 03 08:54:20.017730 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:20.017661    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00bab831-c667-415d-b6b0-81fff4d57762-tmp-dir\") pod \"simulator-w4-18f9da-kserve-66dd7c494b-x4zlh\" (UID: \"00bab831-c667-415d-b6b0-81fff4d57762\") " pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh"
Nov 03 08:54:20.021125 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:20.021099    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00bab831-c667-415d-b6b0-81fff4d57762-dshm\") pod \"simulator-w4-18f9da-kserve-66dd7c494b-x4zlh\" (UID: \"00bab831-c667-415d-b6b0-81fff4d57762\") " pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh"
Nov 03 08:54:20.021400 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:20.021379    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00bab831-c667-415d-b6b0-81fff4d57762-tls-certs\") pod \"simulator-w4-18f9da-kserve-66dd7c494b-x4zlh\" (UID: \"00bab831-c667-415d-b6b0-81fff4d57762\") " pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh"
Nov 03 08:54:20.075628 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:20.075600    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqjlv\" (UniqueName: \"kubernetes.io/projected/00bab831-c667-415d-b6b0-81fff4d57762-kube-api-access-bqjlv\") pod \"simulator-w4-18f9da-kserve-66dd7c494b-x4zlh\" (UID: \"00bab831-c667-415d-b6b0-81fff4d57762\") " pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh"
Nov 03 08:54:20.370122 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:20.370100    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh"
Nov 03 08:54:20.742726 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:20.742700    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh"]
Nov 03 08:54:20.744297 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:54:20.744275    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00bab831_c667_415d_b6b0_81fff4d57762.slice/crio-604db5aa29f63682b2415a9fc345178b03517c82574178fe585aa1a4b02d537d WatchSource:0}: Error finding container 604db5aa29f63682b2415a9fc345178b03517c82574178fe585aa1a4b02d537d: Status 404 returned error can't find the container with id 604db5aa29f63682b2415a9fc345178b03517c82574178fe585aa1a4b02d537d
Nov 03 08:54:20.774897 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:20.774872    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh" event={"ID":"00bab831-c667-415d-b6b0-81fff4d57762","Type":"ContainerStarted","Data":"604db5aa29f63682b2415a9fc345178b03517c82574178fe585aa1a4b02d537d"}
Nov 03 08:54:21.781487 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:21.781456    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh" event={"ID":"00bab831-c667-415d-b6b0-81fff4d57762","Type":"ContainerStarted","Data":"3a357ddbfa22890aeeb1111586d18167a593964ad6d736077e6169c86223dbcf"}
Nov 03 08:54:21.904001 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:21.903936    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh" podStartSLOduration=2.903922115 podStartE2EDuration="2.903922115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:54:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:54:21.902117369 +0000 UTC m=+886.171180821" watchObservedRunningTime="2026-11-03 08:54:21.903922115 +0000 UTC m=+886.172985594"
Nov 03 08:54:22.370459 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:22.370431    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh"
Nov 03 08:54:24.186541 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:24.186505    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 08:54:30.371150 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:30.371118    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh"
Nov 03 08:54:30.383464 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:30.383438    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh"
Nov 03 08:54:30.847524 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:30.847439    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh"
Nov 03 08:54:35.975968 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:35.975924    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 08:54:35.976709 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:35.976055    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 08:54:45.137518 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:45.137482    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd"]
Nov 03 08:54:45.141153 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:45.141136    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd"
Nov 03 08:54:45.215230 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:45.215197    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd"]
Nov 03 08:54:45.243686 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:45.243656    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf4cf2fc-612a-4f5c-a596-543036dde7e3-model-cache\") pod \"simulator-w2-77207e-kserve-74458c56fc-bssxd\" (UID: \"cf4cf2fc-612a-4f5c-a596-543036dde7e3\") " pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd"
Nov 03 08:54:45.243840 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:45.243716    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf4cf2fc-612a-4f5c-a596-543036dde7e3-tls-certs\") pod \"simulator-w2-77207e-kserve-74458c56fc-bssxd\" (UID: \"cf4cf2fc-612a-4f5c-a596-543036dde7e3\") " pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd"
Nov 03 08:54:45.243840 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:45.243804    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf4cf2fc-612a-4f5c-a596-543036dde7e3-home\") pod \"simulator-w2-77207e-kserve-74458c56fc-bssxd\" (UID: \"cf4cf2fc-612a-4f5c-a596-543036dde7e3\") " pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd"
Nov 03 08:54:45.243913 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:45.243863    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf4cf2fc-612a-4f5c-a596-543036dde7e3-dshm\") pod \"simulator-w2-77207e-kserve-74458c56fc-bssxd\" (UID: \"cf4cf2fc-612a-4f5c-a596-543036dde7e3\") " pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd"
Nov 03 08:54:45.243913 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:45.243906    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mnr9\" (UniqueName: \"kubernetes.io/projected/cf4cf2fc-612a-4f5c-a596-543036dde7e3-kube-api-access-6mnr9\") pod \"simulator-w2-77207e-kserve-74458c56fc-bssxd\" (UID: \"cf4cf2fc-612a-4f5c-a596-543036dde7e3\") " pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd"
Nov 03 08:54:45.244012 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:45.243926    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf4cf2fc-612a-4f5c-a596-543036dde7e3-tmp-dir\") pod \"simulator-w2-77207e-kserve-74458c56fc-bssxd\" (UID: \"cf4cf2fc-612a-4f5c-a596-543036dde7e3\") " pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd"
Nov 03 08:54:45.247591 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:45.247570    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-77207e\"/\"default-dockercfg-czmfl\""
Nov 03 08:54:45.247591 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:45.247577    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-77207e\"/\"kube-root-ca.crt\""
Nov 03 08:54:45.247680 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:45.247575    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-77207e\"/\"openshift-service-ca.crt\""
Nov 03 08:54:45.247955 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:45.247929    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-77207e\"/\"simulator-w2-77207e-kserve-self-signed-certs\""
Nov 03 08:54:45.345471 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:45.345443    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mnr9\" (UniqueName: \"kubernetes.io/projected/cf4cf2fc-612a-4f5c-a596-543036dde7e3-kube-api-access-6mnr9\") pod \"simulator-w2-77207e-kserve-74458c56fc-bssxd\" (UID: \"cf4cf2fc-612a-4f5c-a596-543036dde7e3\") " pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd"
Nov 03 08:54:45.345640 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:45.345494    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf4cf2fc-612a-4f5c-a596-543036dde7e3-tmp-dir\") pod \"simulator-w2-77207e-kserve-74458c56fc-bssxd\" (UID: \"cf4cf2fc-612a-4f5c-a596-543036dde7e3\") " pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd"
Nov 03 08:54:45.345640 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:45.345573    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf4cf2fc-612a-4f5c-a596-543036dde7e3-model-cache\") pod \"simulator-w2-77207e-kserve-74458c56fc-bssxd\" (UID: \"cf4cf2fc-612a-4f5c-a596-543036dde7e3\") " pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd"
Nov 03 08:54:45.345640 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:45.345613    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf4cf2fc-612a-4f5c-a596-543036dde7e3-tls-certs\") pod \"simulator-w2-77207e-kserve-74458c56fc-bssxd\" (UID: \"cf4cf2fc-612a-4f5c-a596-543036dde7e3\") " pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd"
Nov 03 08:54:45.345640 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:45.345632    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf4cf2fc-612a-4f5c-a596-543036dde7e3-home\") pod \"simulator-w2-77207e-kserve-74458c56fc-bssxd\" (UID: \"cf4cf2fc-612a-4f5c-a596-543036dde7e3\") " pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd"
Nov 03 08:54:45.345877 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:45.345661    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf4cf2fc-612a-4f5c-a596-543036dde7e3-dshm\") pod \"simulator-w2-77207e-kserve-74458c56fc-bssxd\" (UID: \"cf4cf2fc-612a-4f5c-a596-543036dde7e3\") " pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd"
Nov 03 08:54:45.346017 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:45.345995    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf4cf2fc-612a-4f5c-a596-543036dde7e3-tmp-dir\") pod \"simulator-w2-77207e-kserve-74458c56fc-bssxd\" (UID: \"cf4cf2fc-612a-4f5c-a596-543036dde7e3\") " pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd"
Nov 03 08:54:45.346098 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:45.345995    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf4cf2fc-612a-4f5c-a596-543036dde7e3-model-cache\") pod \"simulator-w2-77207e-kserve-74458c56fc-bssxd\" (UID: \"cf4cf2fc-612a-4f5c-a596-543036dde7e3\") " pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd"
Nov 03 08:54:45.346098 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:45.346079    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf4cf2fc-612a-4f5c-a596-543036dde7e3-home\") pod \"simulator-w2-77207e-kserve-74458c56fc-bssxd\" (UID: \"cf4cf2fc-612a-4f5c-a596-543036dde7e3\") " pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd"
Nov 03 08:54:45.348126 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:45.348103    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf4cf2fc-612a-4f5c-a596-543036dde7e3-dshm\") pod \"simulator-w2-77207e-kserve-74458c56fc-bssxd\" (UID: \"cf4cf2fc-612a-4f5c-a596-543036dde7e3\") " pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd"
Nov 03 08:54:45.348412 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:45.348381    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf4cf2fc-612a-4f5c-a596-543036dde7e3-tls-certs\") pod \"simulator-w2-77207e-kserve-74458c56fc-bssxd\" (UID: \"cf4cf2fc-612a-4f5c-a596-543036dde7e3\") " pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd"
Nov 03 08:54:45.408770 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:45.408697    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mnr9\" (UniqueName: \"kubernetes.io/projected/cf4cf2fc-612a-4f5c-a596-543036dde7e3-kube-api-access-6mnr9\") pod \"simulator-w2-77207e-kserve-74458c56fc-bssxd\" (UID: \"cf4cf2fc-612a-4f5c-a596-543036dde7e3\") " pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd"
Nov 03 08:54:45.452856 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:45.452823    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd"
Nov 03 08:54:45.830773 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:45.830743    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd"]
Nov 03 08:54:45.832689 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:54:45.832653    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf4cf2fc_612a_4f5c_a596_543036dde7e3.slice/crio-457f4e80e89340a0f74614bb47b2cec7be2aedc8af3a676ef77359e897644048 WatchSource:0}: Error finding container 457f4e80e89340a0f74614bb47b2cec7be2aedc8af3a676ef77359e897644048: Status 404 returned error can't find the container with id 457f4e80e89340a0f74614bb47b2cec7be2aedc8af3a676ef77359e897644048
Nov 03 08:54:45.906482 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:45.906447    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd" event={"ID":"cf4cf2fc-612a-4f5c-a596-543036dde7e3","Type":"ContainerStarted","Data":"1a185f8717c5d7f7d5f232ad5e2c83487feb26ce7edb70a70b31dcbe11f71cfd"}
Nov 03 08:54:45.906596 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:45.906498    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd" event={"ID":"cf4cf2fc-612a-4f5c-a596-543036dde7e3","Type":"ContainerStarted","Data":"457f4e80e89340a0f74614bb47b2cec7be2aedc8af3a676ef77359e897644048"}
Nov 03 08:54:46.023910 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:46.023859    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd" podStartSLOduration=1.02384516 podStartE2EDuration="1.02384516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:54:46.023082069 +0000 UTC m=+910.292145504" watchObservedRunningTime="2026-11-03 08:54:46.02384516 +0000 UTC m=+910.292908612"
Nov 03 08:54:46.453471 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:46.453441    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd"
Nov 03 08:54:46.936628 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:54:46.936596    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:54:47.649049 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:47.649009    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"]
Nov 03 08:54:47.654085 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:47.654060    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"
Nov 03 08:54:47.758263 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:47.758233    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-98d5dd-openshift-default-dockercfg-dt94n\""
Nov 03 08:54:47.760215 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:47.760193    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"]
Nov 03 08:54:47.766566 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:47.766542    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9f837be-e9a7-4781-bb0d-8b7a50f63555-workload-certs\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66\" (UID: \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"
Nov 03 08:54:47.766675 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:47.766576    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9f837be-e9a7-4781-bb0d-8b7a50f63555-istio-data\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66\" (UID: \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"
Nov 03 08:54:47.766675 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:47.766611    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9f837be-e9a7-4781-bb0d-8b7a50f63555-workload-socket\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66\" (UID: \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"
Nov 03 08:54:47.766770 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:47.766704    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9f837be-e9a7-4781-bb0d-8b7a50f63555-istio-envoy\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66\" (UID: \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"
Nov 03 08:54:47.766770 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:47.766760    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9f837be-e9a7-4781-bb0d-8b7a50f63555-istio-token\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66\" (UID: \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"
Nov 03 08:54:47.766852 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:47.766832    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9f837be-e9a7-4781-bb0d-8b7a50f63555-credential-socket\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66\" (UID: \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"
Nov 03 08:54:47.766919 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:47.766899    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9f837be-e9a7-4781-bb0d-8b7a50f63555-istiod-ca-cert\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66\" (UID: \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"
Nov 03 08:54:47.766998 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:47.766972    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9f837be-e9a7-4781-bb0d-8b7a50f63555-istio-podinfo\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66\" (UID: \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"
Nov 03 08:54:47.767061 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:47.767045    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dht5v\" (UniqueName: \"kubernetes.io/projected/e9f837be-e9a7-4781-bb0d-8b7a50f63555-kube-api-access-dht5v\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66\" (UID: \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"
Nov 03 08:54:47.867590 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:47.867566    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dht5v\" (UniqueName: \"kubernetes.io/projected/e9f837be-e9a7-4781-bb0d-8b7a50f63555-kube-api-access-dht5v\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66\" (UID: \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"
Nov 03 08:54:47.867752 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:47.867622    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9f837be-e9a7-4781-bb0d-8b7a50f63555-workload-certs\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66\" (UID: \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"
Nov 03 08:54:47.867752 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:47.867646    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9f837be-e9a7-4781-bb0d-8b7a50f63555-istio-data\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66\" (UID: \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"
Nov 03 08:54:47.867752 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:47.867667    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9f837be-e9a7-4781-bb0d-8b7a50f63555-workload-socket\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66\" (UID: \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"
Nov 03 08:54:47.867752 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:47.867689    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9f837be-e9a7-4781-bb0d-8b7a50f63555-istio-envoy\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66\" (UID: \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"
Nov 03 08:54:47.867752 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:47.867715    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9f837be-e9a7-4781-bb0d-8b7a50f63555-istio-token\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66\" (UID: \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"
Nov 03 08:54:47.868065 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:47.867794    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9f837be-e9a7-4781-bb0d-8b7a50f63555-credential-socket\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66\" (UID: \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"
Nov 03 08:54:47.868065 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:47.867822    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9f837be-e9a7-4781-bb0d-8b7a50f63555-istiod-ca-cert\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66\" (UID: \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"
Nov 03 08:54:47.868065 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:47.867854    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9f837be-e9a7-4781-bb0d-8b7a50f63555-istio-podinfo\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66\" (UID: \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"
Nov 03 08:54:47.868232 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:47.868107    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9f837be-e9a7-4781-bb0d-8b7a50f63555-workload-certs\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66\" (UID: \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"
Nov 03 08:54:47.868232 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:47.868174    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9f837be-e9a7-4781-bb0d-8b7a50f63555-istio-data\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66\" (UID: \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"
Nov 03 08:54:47.868340 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:47.868243    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9f837be-e9a7-4781-bb0d-8b7a50f63555-workload-socket\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66\" (UID: \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"
Nov 03 08:54:47.868500 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:47.868484    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9f837be-e9a7-4781-bb0d-8b7a50f63555-credential-socket\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66\" (UID: \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"
Nov 03 08:54:47.868655 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:47.868638    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9f837be-e9a7-4781-bb0d-8b7a50f63555-istiod-ca-cert\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66\" (UID: \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"
Nov 03 08:54:47.870070 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:47.870053    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9f837be-e9a7-4781-bb0d-8b7a50f63555-istio-envoy\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66\" (UID: \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"
Nov 03 08:54:47.870403 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:47.870383    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9f837be-e9a7-4781-bb0d-8b7a50f63555-istio-podinfo\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66\" (UID: \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"
Nov 03 08:54:47.927158 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:47.927097    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9f837be-e9a7-4781-bb0d-8b7a50f63555-istio-token\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66\" (UID: \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"
Nov 03 08:54:47.927284 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:47.927182    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dht5v\" (UniqueName: \"kubernetes.io/projected/e9f837be-e9a7-4781-bb0d-8b7a50f63555-kube-api-access-dht5v\") pod \"e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66\" (UID: \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\") " pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"
Nov 03 08:54:47.968164 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:47.968139    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"
Nov 03 08:54:48.391194 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:48.391171    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"]
Nov 03 08:54:48.392962 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:54:48.392921    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9f837be_e9a7_4781_bb0d_8b7a50f63555.slice/crio-ec5758cf3e2a8831baa6bcd4e7f8344a774df7054e4293608a025144f1bc4c10 WatchSource:0}: Error finding container ec5758cf3e2a8831baa6bcd4e7f8344a774df7054e4293608a025144f1bc4c10: Status 404 returned error can't find the container with id ec5758cf3e2a8831baa6bcd4e7f8344a774df7054e4293608a025144f1bc4c10
Nov 03 08:54:48.395322 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:48.395281    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 08:54:48.395421 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:48.395343    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 08:54:48.395421 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:48.395374    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 08:54:48.925011 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:48.924975    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66" event={"ID":"e9f837be-e9a7-4781-bb0d-8b7a50f63555","Type":"ContainerStarted","Data":"b61b53e74603e54735baf74ef1b15bb2d57990d04f0af046aef41a16a84bd753"}
Nov 03 08:54:48.925602 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:48.925576    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66" event={"ID":"e9f837be-e9a7-4781-bb0d-8b7a50f63555","Type":"ContainerStarted","Data":"ec5758cf3e2a8831baa6bcd4e7f8344a774df7054e4293608a025144f1bc4c10"}
Nov 03 08:54:48.969144 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:48.969102    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"
Nov 03 08:54:49.046660 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:49.046603    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66" podStartSLOduration=2.046587129 podStartE2EDuration="2.046587129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:54:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:54:49.045376661 +0000 UTC m=+913.314440112" watchObservedRunningTime="2026-11-03 08:54:49.046587129 +0000 UTC m=+913.315650563"
Nov 03 08:54:49.970325 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:49.970293    4473 patch_prober.go:29] interesting pod/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 03 08:54:49.970732 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:49.970346    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66" podUID="e9f837be-e9a7-4781-bb0d-8b7a50f63555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 03 08:54:50.969152 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:50.969119    4473 patch_prober.go:29] interesting pod/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 03 08:54:50.969334 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:50.969166    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66" podUID="e9f837be-e9a7-4781-bb0d-8b7a50f63555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 03 08:54:51.135382 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:51.135343    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 08:54:51.968930 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:51.968899    4473 patch_prober.go:29] interesting pod/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 03 08:54:51.969151 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:51.969010    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66" podUID="e9f837be-e9a7-4781-bb0d-8b7a50f63555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 03 08:54:52.968441 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:52.968411    4473 patch_prober.go:29] interesting pod/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 03 08:54:52.968832 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:52.968458    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66" podUID="e9f837be-e9a7-4781-bb0d-8b7a50f63555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 03 08:54:53.969158 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:53.969122    4473 patch_prober.go:29] interesting pod/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 03 08:54:53.969633 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:53.969190    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66" podUID="e9f837be-e9a7-4781-bb0d-8b7a50f63555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 03 08:54:55.097098 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:55.097061    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"
Nov 03 08:54:55.097617 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:55.097501    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"
Nov 03 08:54:55.098505 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:55.098452    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"
Nov 03 08:54:55.453750 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:55.453725    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd"
Nov 03 08:54:55.466457 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:55.466427    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd"
Nov 03 08:54:55.988645 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:54:55.988619    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd"
Nov 03 08:55:04.146025 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:04.145992    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d76c0807-gt9d5"]
Nov 03 08:55:04.149627 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:04.149610    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d76c0807-gt9d5"
Nov 03 08:55:04.207792 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:04.207759    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d76c0807-gt9d5"]
Nov 03 08:55:04.255103 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:04.255070    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:55:04.255269 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:04.255111    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:55:04.255269 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:04.255118    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:55:04.255269 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:04.255189    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 08:55:04.259531 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:04.259507    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4dkd\" (UniqueName: \"kubernetes.io/projected/490bbf6e-ce79-48de-8357-eb0f1c19055e-kube-api-access-d4dkd\") pod \"maas-api-revoke-sub-d76c0807-gt9d5\" (UID: \"490bbf6e-ce79-48de-8357-eb0f1c19055e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d76c0807-gt9d5"
Nov 03 08:55:04.259661 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:04.259544    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/490bbf6e-ce79-48de-8357-eb0f1c19055e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d76c0807-gt9d5\" (UID: \"490bbf6e-ce79-48de-8357-eb0f1c19055e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d76c0807-gt9d5"
Nov 03 08:55:04.259724 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:04.259697    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/490bbf6e-ce79-48de-8357-eb0f1c19055e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d76c0807-gt9d5\" (UID: \"490bbf6e-ce79-48de-8357-eb0f1c19055e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d76c0807-gt9d5"
Nov 03 08:55:04.360713 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:04.360678    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4dkd\" (UniqueName: \"kubernetes.io/projected/490bbf6e-ce79-48de-8357-eb0f1c19055e-kube-api-access-d4dkd\") pod \"maas-api-revoke-sub-d76c0807-gt9d5\" (UID: \"490bbf6e-ce79-48de-8357-eb0f1c19055e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d76c0807-gt9d5"
Nov 03 08:55:04.360713 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:04.360715    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/490bbf6e-ce79-48de-8357-eb0f1c19055e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d76c0807-gt9d5\" (UID: \"490bbf6e-ce79-48de-8357-eb0f1c19055e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d76c0807-gt9d5"
Nov 03 08:55:04.361009 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:04.360831    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/490bbf6e-ce79-48de-8357-eb0f1c19055e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d76c0807-gt9d5\" (UID: \"490bbf6e-ce79-48de-8357-eb0f1c19055e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d76c0807-gt9d5"
Nov 03 08:55:04.361476 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:04.361452    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/490bbf6e-ce79-48de-8357-eb0f1c19055e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d76c0807-gt9d5\" (UID: \"490bbf6e-ce79-48de-8357-eb0f1c19055e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d76c0807-gt9d5"
Nov 03 08:55:04.363328 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:04.363311    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/490bbf6e-ce79-48de-8357-eb0f1c19055e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d76c0807-gt9d5\" (UID: \"490bbf6e-ce79-48de-8357-eb0f1c19055e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d76c0807-gt9d5"
Nov 03 08:55:04.420542 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:04.420483    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4dkd\" (UniqueName: \"kubernetes.io/projected/490bbf6e-ce79-48de-8357-eb0f1c19055e-kube-api-access-d4dkd\") pod \"maas-api-revoke-sub-d76c0807-gt9d5\" (UID: \"490bbf6e-ce79-48de-8357-eb0f1c19055e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d76c0807-gt9d5"
Nov 03 08:55:04.463520 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:04.463485    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d76c0807-gt9d5"
Nov 03 08:55:04.834885 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:04.834864    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d76c0807-gt9d5"]
Nov 03 08:55:04.836596 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:55:04.836562    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod490bbf6e_ce79_48de_8357_eb0f1c19055e.slice/crio-0772d33fb2f496a9ba8b3c358ffe431a4f583583e2a3092d246b5591e8c3ee1b WatchSource:0}: Error finding container 0772d33fb2f496a9ba8b3c358ffe431a4f583583e2a3092d246b5591e8c3ee1b: Status 404 returned error can't find the container with id 0772d33fb2f496a9ba8b3c358ffe431a4f583583e2a3092d246b5591e8c3ee1b
Nov 03 08:55:05.042022 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:05.041989    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d76c0807-gt9d5" event={"ID":"490bbf6e-ce79-48de-8357-eb0f1c19055e","Type":"ContainerStarted","Data":"0772d33fb2f496a9ba8b3c358ffe431a4f583583e2a3092d246b5591e8c3ee1b"}
Nov 03 08:55:06.052747 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:06.052631    4473 generic.go:356] "Generic (PLEG): container finished" podID="490bbf6e-ce79-48de-8357-eb0f1c19055e" containerID="90044752bc2d8c52df0b0f0c3a9828cc984ad63ae631813016a29b5aeaf96f55" exitCode=0
Nov 03 08:55:06.053160 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:06.052752    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d76c0807-gt9d5" event={"ID":"490bbf6e-ce79-48de-8357-eb0f1c19055e","Type":"ContainerDied","Data":"90044752bc2d8c52df0b0f0c3a9828cc984ad63ae631813016a29b5aeaf96f55"}
Nov 03 08:55:07.191956 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:07.191918    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d76c0807-gt9d5"
Nov 03 08:55:07.287707 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:07.287674    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/490bbf6e-ce79-48de-8357-eb0f1c19055e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/490bbf6e-ce79-48de-8357-eb0f1c19055e-maas-api-openshift-ca\") pod \"490bbf6e-ce79-48de-8357-eb0f1c19055e\" (UID: \"490bbf6e-ce79-48de-8357-eb0f1c19055e\") "
Nov 03 08:55:07.287868 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:07.287727    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/490bbf6e-ce79-48de-8357-eb0f1c19055e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/490bbf6e-ce79-48de-8357-eb0f1c19055e-maas-api-xks-ca\") pod \"490bbf6e-ce79-48de-8357-eb0f1c19055e\" (UID: \"490bbf6e-ce79-48de-8357-eb0f1c19055e\") "
Nov 03 08:55:07.287868 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:07.287757    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/490bbf6e-ce79-48de-8357-eb0f1c19055e-kube-api-access-d4dkd\" (UniqueName: \"kubernetes.io/projected/490bbf6e-ce79-48de-8357-eb0f1c19055e-kube-api-access-d4dkd\") pod \"490bbf6e-ce79-48de-8357-eb0f1c19055e\" (UID: \"490bbf6e-ce79-48de-8357-eb0f1c19055e\") "
Nov 03 08:55:07.288072 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:07.288050    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/490bbf6e-ce79-48de-8357-eb0f1c19055e-maas-api-openshift-ca" pod "490bbf6e-ce79-48de-8357-eb0f1c19055e" (UID: "490bbf6e-ce79-48de-8357-eb0f1c19055e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:55:07.290058 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:07.290036    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/490bbf6e-ce79-48de-8357-eb0f1c19055e-maas-api-xks-ca" pod "490bbf6e-ce79-48de-8357-eb0f1c19055e" (UID: "490bbf6e-ce79-48de-8357-eb0f1c19055e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:55:07.290154 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:07.290097    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/490bbf6e-ce79-48de-8357-eb0f1c19055e-kube-api-access-d4dkd" pod "490bbf6e-ce79-48de-8357-eb0f1c19055e" (UID: "490bbf6e-ce79-48de-8357-eb0f1c19055e"). InnerVolumeSpecName "kube-api-access-d4dkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:55:07.389175 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:07.389144    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/490bbf6e-ce79-48de-8357-eb0f1c19055e-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:55:07.389175 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:07.389174    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/490bbf6e-ce79-48de-8357-eb0f1c19055e-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:55:07.389361 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:07.389186    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4dkd\" (UniqueName: \"kubernetes.io/projected/490bbf6e-ce79-48de-8357-eb0f1c19055e-kube-api-access-d4dkd\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:55:08.064567 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:08.064536    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d76c0807-gt9d5"
Nov 03 08:55:08.064772 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:08.064580    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d76c0807-gt9d5" event={"ID":"490bbf6e-ce79-48de-8357-eb0f1c19055e","Type":"ContainerDied","Data":"0772d33fb2f496a9ba8b3c358ffe431a4f583583e2a3092d246b5591e8c3ee1b"}
Nov 03 08:55:08.064772 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:08.064628    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0772d33fb2f496a9ba8b3c358ffe431a4f583583e2a3092d246b5591e8c3ee1b"
Nov 03 08:55:09.127473 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:09.127442    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 08:55:14.858577 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:14.858542    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm"]
Nov 03 08:55:14.859154 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:14.859135    4473 cpu_manager.go:414] "containerMap: removing container" podUID="490bbf6e-ce79-48de-8357-eb0f1c19055e" containerName="revoke-keys"
Nov 03 08:55:14.859244 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:14.859157    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="490bbf6e-ce79-48de-8357-eb0f1c19055e" containerName="revoke-keys"
Nov 03 08:55:14.859310 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:14.859282    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="490bbf6e-ce79-48de-8357-eb0f1c19055e" containerName="revoke-keys"
Nov 03 08:55:14.863685 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:14.863662    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm"
Nov 03 08:55:14.933770 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:14.933738    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm"]
Nov 03 08:55:14.957113 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:14.957081    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60b1d7ef-16c6-4b6a-973b-4a251593eac7-model-cache\") pod \"premium-w4-18f9da-kserve-5d68844675-5jpkm\" (UID: \"60b1d7ef-16c6-4b6a-973b-4a251593eac7\") " pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm"
Nov 03 08:55:14.957259 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:14.957132    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60b1d7ef-16c6-4b6a-973b-4a251593eac7-dshm\") pod \"premium-w4-18f9da-kserve-5d68844675-5jpkm\" (UID: \"60b1d7ef-16c6-4b6a-973b-4a251593eac7\") " pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm"
Nov 03 08:55:14.957259 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:14.957198    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtwkq\" (UniqueName: \"kubernetes.io/projected/60b1d7ef-16c6-4b6a-973b-4a251593eac7-kube-api-access-gtwkq\") pod \"premium-w4-18f9da-kserve-5d68844675-5jpkm\" (UID: \"60b1d7ef-16c6-4b6a-973b-4a251593eac7\") " pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm"
Nov 03 08:55:14.957259 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:14.957236    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60b1d7ef-16c6-4b6a-973b-4a251593eac7-home\") pod \"premium-w4-18f9da-kserve-5d68844675-5jpkm\" (UID: \"60b1d7ef-16c6-4b6a-973b-4a251593eac7\") " pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm"
Nov 03 08:55:14.957410 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:14.957264    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60b1d7ef-16c6-4b6a-973b-4a251593eac7-tls-certs\") pod \"premium-w4-18f9da-kserve-5d68844675-5jpkm\" (UID: \"60b1d7ef-16c6-4b6a-973b-4a251593eac7\") " pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm"
Nov 03 08:55:14.957410 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:14.957300    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60b1d7ef-16c6-4b6a-973b-4a251593eac7-tmp-dir\") pod \"premium-w4-18f9da-kserve-5d68844675-5jpkm\" (UID: \"60b1d7ef-16c6-4b6a-973b-4a251593eac7\") " pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm"
Nov 03 08:55:14.968289 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:14.968266    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-18f9da\"/\"premium-w4-18f9da-kserve-self-signed-certs\""
Nov 03 08:55:15.058492 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:15.058460    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60b1d7ef-16c6-4b6a-973b-4a251593eac7-model-cache\") pod \"premium-w4-18f9da-kserve-5d68844675-5jpkm\" (UID: \"60b1d7ef-16c6-4b6a-973b-4a251593eac7\") " pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm"
Nov 03 08:55:15.058670 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:15.058509    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60b1d7ef-16c6-4b6a-973b-4a251593eac7-dshm\") pod \"premium-w4-18f9da-kserve-5d68844675-5jpkm\" (UID: \"60b1d7ef-16c6-4b6a-973b-4a251593eac7\") " pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm"
Nov 03 08:55:15.058670 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:15.058528    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtwkq\" (UniqueName: \"kubernetes.io/projected/60b1d7ef-16c6-4b6a-973b-4a251593eac7-kube-api-access-gtwkq\") pod \"premium-w4-18f9da-kserve-5d68844675-5jpkm\" (UID: \"60b1d7ef-16c6-4b6a-973b-4a251593eac7\") " pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm"
Nov 03 08:55:15.058753 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:15.058660    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60b1d7ef-16c6-4b6a-973b-4a251593eac7-home\") pod \"premium-w4-18f9da-kserve-5d68844675-5jpkm\" (UID: \"60b1d7ef-16c6-4b6a-973b-4a251593eac7\") " pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm"
Nov 03 08:55:15.058753 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:15.058716    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60b1d7ef-16c6-4b6a-973b-4a251593eac7-tls-certs\") pod \"premium-w4-18f9da-kserve-5d68844675-5jpkm\" (UID: \"60b1d7ef-16c6-4b6a-973b-4a251593eac7\") " pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm"
Nov 03 08:55:15.058845 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:15.058752    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60b1d7ef-16c6-4b6a-973b-4a251593eac7-tmp-dir\") pod \"premium-w4-18f9da-kserve-5d68844675-5jpkm\" (UID: \"60b1d7ef-16c6-4b6a-973b-4a251593eac7\") " pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm"
Nov 03 08:55:15.059106 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:15.059084    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60b1d7ef-16c6-4b6a-973b-4a251593eac7-model-cache\") pod \"premium-w4-18f9da-kserve-5d68844675-5jpkm\" (UID: \"60b1d7ef-16c6-4b6a-973b-4a251593eac7\") " pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm"
Nov 03 08:55:15.059221 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:15.059124    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60b1d7ef-16c6-4b6a-973b-4a251593eac7-home\") pod \"premium-w4-18f9da-kserve-5d68844675-5jpkm\" (UID: \"60b1d7ef-16c6-4b6a-973b-4a251593eac7\") " pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm"
Nov 03 08:55:15.059221 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:15.059177    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60b1d7ef-16c6-4b6a-973b-4a251593eac7-tmp-dir\") pod \"premium-w4-18f9da-kserve-5d68844675-5jpkm\" (UID: \"60b1d7ef-16c6-4b6a-973b-4a251593eac7\") " pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm"
Nov 03 08:55:15.060935 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:15.060917    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60b1d7ef-16c6-4b6a-973b-4a251593eac7-dshm\") pod \"premium-w4-18f9da-kserve-5d68844675-5jpkm\" (UID: \"60b1d7ef-16c6-4b6a-973b-4a251593eac7\") " pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm"
Nov 03 08:55:15.061433 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:15.061408    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60b1d7ef-16c6-4b6a-973b-4a251593eac7-tls-certs\") pod \"premium-w4-18f9da-kserve-5d68844675-5jpkm\" (UID: \"60b1d7ef-16c6-4b6a-973b-4a251593eac7\") " pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm"
Nov 03 08:55:15.120752 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:15.120703    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtwkq\" (UniqueName: \"kubernetes.io/projected/60b1d7ef-16c6-4b6a-973b-4a251593eac7-kube-api-access-gtwkq\") pod \"premium-w4-18f9da-kserve-5d68844675-5jpkm\" (UID: \"60b1d7ef-16c6-4b6a-973b-4a251593eac7\") " pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm"
Nov 03 08:55:15.175507 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:15.175463    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm"
Nov 03 08:55:15.584158 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:15.584133    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm"]
Nov 03 08:55:15.585793 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:55:15.585766    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60b1d7ef_16c6_4b6a_973b_4a251593eac7.slice/crio-a8e5579320c4ee4c2aa165782f3dae107b74cac51a871041a4fa8590c65d12d3 WatchSource:0}: Error finding container a8e5579320c4ee4c2aa165782f3dae107b74cac51a871041a4fa8590c65d12d3: Status 404 returned error can't find the container with id a8e5579320c4ee4c2aa165782f3dae107b74cac51a871041a4fa8590c65d12d3
Nov 03 08:55:16.106561 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:16.106524    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm" event={"ID":"60b1d7ef-16c6-4b6a-973b-4a251593eac7","Type":"ContainerStarted","Data":"3744118d3b2020f3e6089947695f142df33af61398f13df40b08c08432f9f451"}
Nov 03 08:55:16.106939 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:16.106569    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm" event={"ID":"60b1d7ef-16c6-4b6a-973b-4a251593eac7","Type":"ContainerStarted","Data":"a8e5579320c4ee4c2aa165782f3dae107b74cac51a871041a4fa8590c65d12d3"}
Nov 03 08:55:16.176000 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:16.175975    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm"
Nov 03 08:55:16.226065 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:16.226018    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm" podStartSLOduration=2.226004384 podStartE2EDuration="2.226004384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:55:16.223524698 +0000 UTC m=+940.492588147" watchObservedRunningTime="2026-11-03 08:55:16.226004384 +0000 UTC m=+940.495067818"
Nov 03 08:55:16.937447 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:55:16.937371    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:55:22.219512 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:22.219482    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-689cb311-wkgx9"]
Nov 03 08:55:22.225168 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:22.225137    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-689cb311-wkgx9"
Nov 03 08:55:22.280867 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:22.280831    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-689cb311-wkgx9"]
Nov 03 08:55:22.328803 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:22.328770    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:55:22.328982 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:22.328820    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:55:22.328982 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:22.328863    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 08:55:22.329117 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:22.329093    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61c2cf80-4442-4226-8b29-ff9ec8245461-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-689cb311-wkgx9\" (UID: \"61c2cf80-4442-4226-8b29-ff9ec8245461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-689cb311-wkgx9"
Nov 03 08:55:22.329201 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:22.329177    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkrls\" (UniqueName: \"kubernetes.io/projected/61c2cf80-4442-4226-8b29-ff9ec8245461-kube-api-access-gkrls\") pod \"maas-api-revoke-sub-689cb311-wkgx9\" (UID: \"61c2cf80-4442-4226-8b29-ff9ec8245461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-689cb311-wkgx9"
Nov 03 08:55:22.329271 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:22.329223    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61c2cf80-4442-4226-8b29-ff9ec8245461-maas-api-xks-ca\") pod \"maas-api-revoke-sub-689cb311-wkgx9\" (UID: \"61c2cf80-4442-4226-8b29-ff9ec8245461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-689cb311-wkgx9"
Nov 03 08:55:22.329271 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:22.329238    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:55:22.430742 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:22.430707    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61c2cf80-4442-4226-8b29-ff9ec8245461-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-689cb311-wkgx9\" (UID: \"61c2cf80-4442-4226-8b29-ff9ec8245461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-689cb311-wkgx9"
Nov 03 08:55:22.430906 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:22.430792    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkrls\" (UniqueName: \"kubernetes.io/projected/61c2cf80-4442-4226-8b29-ff9ec8245461-kube-api-access-gkrls\") pod \"maas-api-revoke-sub-689cb311-wkgx9\" (UID: \"61c2cf80-4442-4226-8b29-ff9ec8245461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-689cb311-wkgx9"
Nov 03 08:55:22.430906 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:22.430822    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61c2cf80-4442-4226-8b29-ff9ec8245461-maas-api-xks-ca\") pod \"maas-api-revoke-sub-689cb311-wkgx9\" (UID: \"61c2cf80-4442-4226-8b29-ff9ec8245461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-689cb311-wkgx9"
Nov 03 08:55:22.431378 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:22.431352    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61c2cf80-4442-4226-8b29-ff9ec8245461-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-689cb311-wkgx9\" (UID: \"61c2cf80-4442-4226-8b29-ff9ec8245461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-689cb311-wkgx9"
Nov 03 08:55:22.433310 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:22.433289    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61c2cf80-4442-4226-8b29-ff9ec8245461-maas-api-xks-ca\") pod \"maas-api-revoke-sub-689cb311-wkgx9\" (UID: \"61c2cf80-4442-4226-8b29-ff9ec8245461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-689cb311-wkgx9"
Nov 03 08:55:22.490313 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:22.490254    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkrls\" (UniqueName: \"kubernetes.io/projected/61c2cf80-4442-4226-8b29-ff9ec8245461-kube-api-access-gkrls\") pod \"maas-api-revoke-sub-689cb311-wkgx9\" (UID: \"61c2cf80-4442-4226-8b29-ff9ec8245461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-689cb311-wkgx9"
Nov 03 08:55:22.540050 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:22.540026    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-689cb311-wkgx9"
Nov 03 08:55:22.915990 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:22.915939    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-689cb311-wkgx9"]
Nov 03 08:55:22.917592 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:55:22.917557    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61c2cf80_4442_4226_8b29_ff9ec8245461.slice/crio-66dfbdf6870ba3aeaae5b3d13abfac1e5477a25deec3b518f72e37865dcf5fcc WatchSource:0}: Error finding container 66dfbdf6870ba3aeaae5b3d13abfac1e5477a25deec3b518f72e37865dcf5fcc: Status 404 returned error can't find the container with id 66dfbdf6870ba3aeaae5b3d13abfac1e5477a25deec3b518f72e37865dcf5fcc
Nov 03 08:55:23.146659 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:23.146626    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-689cb311-wkgx9" event={"ID":"61c2cf80-4442-4226-8b29-ff9ec8245461","Type":"ContainerStarted","Data":"66dfbdf6870ba3aeaae5b3d13abfac1e5477a25deec3b518f72e37865dcf5fcc"}
Nov 03 08:55:24.154062 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:24.154033    4473 generic.go:356] "Generic (PLEG): container finished" podID="61c2cf80-4442-4226-8b29-ff9ec8245461" containerID="9e5d0c9d89c3d73a5f74d91f6952a2b3de64a931653f29b1ed0af16a72f54656" exitCode=0
Nov 03 08:55:24.154444 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:24.154121    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-689cb311-wkgx9" event={"ID":"61c2cf80-4442-4226-8b29-ff9ec8245461","Type":"ContainerDied","Data":"9e5d0c9d89c3d73a5f74d91f6952a2b3de64a931653f29b1ed0af16a72f54656"}
Nov 03 08:55:25.176546 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:25.176512    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm"
Nov 03 08:55:25.191584 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:25.191559    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm"
Nov 03 08:55:25.296499 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:25.296478    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-689cb311-wkgx9"
Nov 03 08:55:25.360652 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:25.360628    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61c2cf80-4442-4226-8b29-ff9ec8245461-kube-api-access-gkrls\" (UniqueName: \"kubernetes.io/projected/61c2cf80-4442-4226-8b29-ff9ec8245461-kube-api-access-gkrls\") pod \"61c2cf80-4442-4226-8b29-ff9ec8245461\" (UID: \"61c2cf80-4442-4226-8b29-ff9ec8245461\") "
Nov 03 08:55:25.360855 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:25.360692    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61c2cf80-4442-4226-8b29-ff9ec8245461-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61c2cf80-4442-4226-8b29-ff9ec8245461-maas-api-xks-ca\") pod \"61c2cf80-4442-4226-8b29-ff9ec8245461\" (UID: \"61c2cf80-4442-4226-8b29-ff9ec8245461\") "
Nov 03 08:55:25.360855 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:25.360779    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61c2cf80-4442-4226-8b29-ff9ec8245461-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61c2cf80-4442-4226-8b29-ff9ec8245461-maas-api-openshift-ca\") pod \"61c2cf80-4442-4226-8b29-ff9ec8245461\" (UID: \"61c2cf80-4442-4226-8b29-ff9ec8245461\") "
Nov 03 08:55:25.361157 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:25.361133    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61c2cf80-4442-4226-8b29-ff9ec8245461-maas-api-openshift-ca" pod "61c2cf80-4442-4226-8b29-ff9ec8245461" (UID: "61c2cf80-4442-4226-8b29-ff9ec8245461"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:55:25.362778 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:25.362757    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61c2cf80-4442-4226-8b29-ff9ec8245461-kube-api-access-gkrls" pod "61c2cf80-4442-4226-8b29-ff9ec8245461" (UID: "61c2cf80-4442-4226-8b29-ff9ec8245461"). InnerVolumeSpecName "kube-api-access-gkrls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:55:25.362845 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:25.362801    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61c2cf80-4442-4226-8b29-ff9ec8245461-maas-api-xks-ca" pod "61c2cf80-4442-4226-8b29-ff9ec8245461" (UID: "61c2cf80-4442-4226-8b29-ff9ec8245461"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:55:25.462246 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:25.462182    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkrls\" (UniqueName: \"kubernetes.io/projected/61c2cf80-4442-4226-8b29-ff9ec8245461-kube-api-access-gkrls\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:55:25.462246 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:25.462208    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61c2cf80-4442-4226-8b29-ff9ec8245461-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:55:25.462246 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:25.462217    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61c2cf80-4442-4226-8b29-ff9ec8245461-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:55:26.164805 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:26.164734    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-689cb311-wkgx9"
Nov 03 08:55:26.164805 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:26.164772    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-689cb311-wkgx9" event={"ID":"61c2cf80-4442-4226-8b29-ff9ec8245461","Type":"ContainerDied","Data":"66dfbdf6870ba3aeaae5b3d13abfac1e5477a25deec3b518f72e37865dcf5fcc"}
Nov 03 08:55:26.165005 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:26.164819    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66dfbdf6870ba3aeaae5b3d13abfac1e5477a25deec3b518f72e37865dcf5fcc"
Nov 03 08:55:26.175989 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:26.175964    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm"
Nov 03 08:55:30.807150 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:30.807118    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 08:55:46.936428 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:55:46.936386    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:55:47.261919 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:47.261843    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 08:55:52.922887 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:52.922858    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27858e00-l4lc5"]
Nov 03 08:55:52.923311 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:52.923269    4473 cpu_manager.go:414] "containerMap: removing container" podUID="61c2cf80-4442-4226-8b29-ff9ec8245461" containerName="revoke-keys"
Nov 03 08:55:52.923311 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:52.923281    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61c2cf80-4442-4226-8b29-ff9ec8245461" containerName="revoke-keys"
Nov 03 08:55:52.923383 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:52.923372    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="61c2cf80-4442-4226-8b29-ff9ec8245461" containerName="revoke-keys"
Nov 03 08:55:52.926281 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:52.926265    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27858e00-l4lc5"
Nov 03 08:55:52.927553 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:52.927521    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bc4f54b-1dc8-4327-9d97-930ca5774ab9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27858e00-l4lc5\" (UID: \"5bc4f54b-1dc8-4327-9d97-930ca5774ab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27858e00-l4lc5"
Nov 03 08:55:52.927682 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:52.927571    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bc4f54b-1dc8-4327-9d97-930ca5774ab9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27858e00-l4lc5\" (UID: \"5bc4f54b-1dc8-4327-9d97-930ca5774ab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27858e00-l4lc5"
Nov 03 08:55:52.927682 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:52.927664    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9469\" (UniqueName: \"kubernetes.io/projected/5bc4f54b-1dc8-4327-9d97-930ca5774ab9-kube-api-access-t9469\") pod \"maas-api-revoke-sub-27858e00-l4lc5\" (UID: \"5bc4f54b-1dc8-4327-9d97-930ca5774ab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27858e00-l4lc5"
Nov 03 08:55:52.982209 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:52.982184    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:55:53.028636 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:53.028607    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bc4f54b-1dc8-4327-9d97-930ca5774ab9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27858e00-l4lc5\" (UID: \"5bc4f54b-1dc8-4327-9d97-930ca5774ab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27858e00-l4lc5"
Nov 03 08:55:53.028776 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:53.028648    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bc4f54b-1dc8-4327-9d97-930ca5774ab9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27858e00-l4lc5\" (UID: \"5bc4f54b-1dc8-4327-9d97-930ca5774ab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27858e00-l4lc5"
Nov 03 08:55:53.028776 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:53.028670    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9469\" (UniqueName: \"kubernetes.io/projected/5bc4f54b-1dc8-4327-9d97-930ca5774ab9-kube-api-access-t9469\") pod \"maas-api-revoke-sub-27858e00-l4lc5\" (UID: \"5bc4f54b-1dc8-4327-9d97-930ca5774ab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27858e00-l4lc5"
Nov 03 08:55:53.031151 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:53.031128    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bc4f54b-1dc8-4327-9d97-930ca5774ab9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27858e00-l4lc5\" (UID: \"5bc4f54b-1dc8-4327-9d97-930ca5774ab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27858e00-l4lc5"
Nov 03 08:55:53.032137 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:53.032123    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:55:53.032512 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:53.032496    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:55:53.032581 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:53.032565    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 08:55:53.034498 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:53.034481    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27858e00-l4lc5"]
Nov 03 08:55:53.040105 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:53.040086    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bc4f54b-1dc8-4327-9d97-930ca5774ab9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27858e00-l4lc5\" (UID: \"5bc4f54b-1dc8-4327-9d97-930ca5774ab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27858e00-l4lc5"
Nov 03 08:55:53.088120 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:53.088095    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9469\" (UniqueName: \"kubernetes.io/projected/5bc4f54b-1dc8-4327-9d97-930ca5774ab9-kube-api-access-t9469\") pod \"maas-api-revoke-sub-27858e00-l4lc5\" (UID: \"5bc4f54b-1dc8-4327-9d97-930ca5774ab9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27858e00-l4lc5"
Nov 03 08:55:53.239292 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:53.239219    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27858e00-l4lc5"
Nov 03 08:55:53.611422 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:53.611401    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27858e00-l4lc5"]
Nov 03 08:55:53.613378 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:55:53.613348    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bc4f54b_1dc8_4327_9d97_930ca5774ab9.slice/crio-a723d6fb49baae1551d76ca05f5e9cded6f6d5c9713f90fdcb57698351fc122d WatchSource:0}: Error finding container a723d6fb49baae1551d76ca05f5e9cded6f6d5c9713f90fdcb57698351fc122d: Status 404 returned error can't find the container with id a723d6fb49baae1551d76ca05f5e9cded6f6d5c9713f90fdcb57698351fc122d
Nov 03 08:55:54.324107 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:54.324028    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27858e00-l4lc5" event={"ID":"5bc4f54b-1dc8-4327-9d97-930ca5774ab9","Type":"ContainerStarted","Data":"5e9ae64e0a5951d47c2ee6766f9b9e09868da6684a87dc34cffebd8f61ac8f0e"}
Nov 03 08:55:54.324107 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:54.324093    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27858e00-l4lc5" event={"ID":"5bc4f54b-1dc8-4327-9d97-930ca5774ab9","Type":"ContainerStarted","Data":"a723d6fb49baae1551d76ca05f5e9cded6f6d5c9713f90fdcb57698351fc122d"}
Nov 03 08:55:54.441503 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:54.441456    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27858e00-l4lc5" podStartSLOduration=2.441442001 podStartE2EDuration="2.441442001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:55:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:55:54.440454274 +0000 UTC m=+978.709517920" watchObservedRunningTime="2026-11-03 08:55:54.441442001 +0000 UTC m=+978.710505434"
Nov 03 08:55:55.334233 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:55.334197    4473 generic.go:356] "Generic (PLEG): container finished" podID="5bc4f54b-1dc8-4327-9d97-930ca5774ab9" containerID="5e9ae64e0a5951d47c2ee6766f9b9e09868da6684a87dc34cffebd8f61ac8f0e" exitCode=0
Nov 03 08:55:55.334670 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:55.334332    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27858e00-l4lc5" event={"ID":"5bc4f54b-1dc8-4327-9d97-930ca5774ab9","Type":"ContainerDied","Data":"5e9ae64e0a5951d47c2ee6766f9b9e09868da6684a87dc34cffebd8f61ac8f0e"}
Nov 03 08:55:56.476092 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:56.476065    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27858e00-l4lc5"
Nov 03 08:55:56.560053 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:56.560024    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bc4f54b-1dc8-4327-9d97-930ca5774ab9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bc4f54b-1dc8-4327-9d97-930ca5774ab9-maas-api-xks-ca\") pod \"5bc4f54b-1dc8-4327-9d97-930ca5774ab9\" (UID: \"5bc4f54b-1dc8-4327-9d97-930ca5774ab9\") "
Nov 03 08:55:56.560229 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:56.560078    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bc4f54b-1dc8-4327-9d97-930ca5774ab9-kube-api-access-t9469\" (UniqueName: \"kubernetes.io/projected/5bc4f54b-1dc8-4327-9d97-930ca5774ab9-kube-api-access-t9469\") pod \"5bc4f54b-1dc8-4327-9d97-930ca5774ab9\" (UID: \"5bc4f54b-1dc8-4327-9d97-930ca5774ab9\") "
Nov 03 08:55:56.560229 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:56.560129    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bc4f54b-1dc8-4327-9d97-930ca5774ab9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bc4f54b-1dc8-4327-9d97-930ca5774ab9-maas-api-openshift-ca\") pod \"5bc4f54b-1dc8-4327-9d97-930ca5774ab9\" (UID: \"5bc4f54b-1dc8-4327-9d97-930ca5774ab9\") "
Nov 03 08:55:56.560500 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:56.560476    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bc4f54b-1dc8-4327-9d97-930ca5774ab9-maas-api-openshift-ca" pod "5bc4f54b-1dc8-4327-9d97-930ca5774ab9" (UID: "5bc4f54b-1dc8-4327-9d97-930ca5774ab9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:55:56.562364 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:56.562334    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bc4f54b-1dc8-4327-9d97-930ca5774ab9-maas-api-xks-ca" pod "5bc4f54b-1dc8-4327-9d97-930ca5774ab9" (UID: "5bc4f54b-1dc8-4327-9d97-930ca5774ab9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:55:56.562364 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:56.562351    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bc4f54b-1dc8-4327-9d97-930ca5774ab9-kube-api-access-t9469" pod "5bc4f54b-1dc8-4327-9d97-930ca5774ab9" (UID: "5bc4f54b-1dc8-4327-9d97-930ca5774ab9"). InnerVolumeSpecName "kube-api-access-t9469". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:55:56.661394 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:56.661369    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bc4f54b-1dc8-4327-9d97-930ca5774ab9-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:55:56.661394 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:56.661394    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9469\" (UniqueName: \"kubernetes.io/projected/5bc4f54b-1dc8-4327-9d97-930ca5774ab9-kube-api-access-t9469\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:55:56.661580 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:56.661409    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bc4f54b-1dc8-4327-9d97-930ca5774ab9-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:55:57.345958 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:57.345913    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27858e00-l4lc5" event={"ID":"5bc4f54b-1dc8-4327-9d97-930ca5774ab9","Type":"ContainerDied","Data":"a723d6fb49baae1551d76ca05f5e9cded6f6d5c9713f90fdcb57698351fc122d"}
Nov 03 08:55:57.345958 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:57.345929    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27858e00-l4lc5"
Nov 03 08:55:57.346159 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:55:57.345978    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a723d6fb49baae1551d76ca05f5e9cded6f6d5c9713f90fdcb57698351fc122d"
Nov 03 08:56:08.243547 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:08.243514    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7639d7e8-zwlsr"]
Nov 03 08:56:08.243979 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:08.243966    4473 cpu_manager.go:414] "containerMap: removing container" podUID="5bc4f54b-1dc8-4327-9d97-930ca5774ab9" containerName="revoke-keys"
Nov 03 08:56:08.244032 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:08.243981    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bc4f54b-1dc8-4327-9d97-930ca5774ab9" containerName="revoke-keys"
Nov 03 08:56:08.244066 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:08.244055    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bc4f54b-1dc8-4327-9d97-930ca5774ab9" containerName="revoke-keys"
Nov 03 08:56:08.247286 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:08.247270    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7639d7e8-zwlsr"
Nov 03 08:56:08.305522 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:08.305490    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7639d7e8-zwlsr"]
Nov 03 08:56:08.352769 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:08.352736    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:56:08.352769 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:08.352749    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:56:08.352979 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:08.352776    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:56:08.352979 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:08.352827    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 08:56:08.380290 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:08.380266    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07d5d4c6-25ee-4574-b508-7c9ef5b3be45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7639d7e8-zwlsr\" (UID: \"07d5d4c6-25ee-4574-b508-7c9ef5b3be45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7639d7e8-zwlsr"
Nov 03 08:56:08.380430 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:08.380329    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhdkt\" (UniqueName: \"kubernetes.io/projected/07d5d4c6-25ee-4574-b508-7c9ef5b3be45-kube-api-access-bhdkt\") pod \"maas-api-revoke-sub-7639d7e8-zwlsr\" (UID: \"07d5d4c6-25ee-4574-b508-7c9ef5b3be45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7639d7e8-zwlsr"
Nov 03 08:56:08.380430 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:08.380407    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07d5d4c6-25ee-4574-b508-7c9ef5b3be45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7639d7e8-zwlsr\" (UID: \"07d5d4c6-25ee-4574-b508-7c9ef5b3be45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7639d7e8-zwlsr"
Nov 03 08:56:08.481703 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:08.481670    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhdkt\" (UniqueName: \"kubernetes.io/projected/07d5d4c6-25ee-4574-b508-7c9ef5b3be45-kube-api-access-bhdkt\") pod \"maas-api-revoke-sub-7639d7e8-zwlsr\" (UID: \"07d5d4c6-25ee-4574-b508-7c9ef5b3be45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7639d7e8-zwlsr"
Nov 03 08:56:08.481877 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:08.481713    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07d5d4c6-25ee-4574-b508-7c9ef5b3be45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7639d7e8-zwlsr\" (UID: \"07d5d4c6-25ee-4574-b508-7c9ef5b3be45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7639d7e8-zwlsr"
Nov 03 08:56:08.481877 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:08.481795    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07d5d4c6-25ee-4574-b508-7c9ef5b3be45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7639d7e8-zwlsr\" (UID: \"07d5d4c6-25ee-4574-b508-7c9ef5b3be45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7639d7e8-zwlsr"
Nov 03 08:56:08.482482 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:08.482458    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07d5d4c6-25ee-4574-b508-7c9ef5b3be45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7639d7e8-zwlsr\" (UID: \"07d5d4c6-25ee-4574-b508-7c9ef5b3be45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7639d7e8-zwlsr"
Nov 03 08:56:08.484360 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:08.484337    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07d5d4c6-25ee-4574-b508-7c9ef5b3be45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7639d7e8-zwlsr\" (UID: \"07d5d4c6-25ee-4574-b508-7c9ef5b3be45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7639d7e8-zwlsr"
Nov 03 08:56:08.544180 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:08.544122    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhdkt\" (UniqueName: \"kubernetes.io/projected/07d5d4c6-25ee-4574-b508-7c9ef5b3be45-kube-api-access-bhdkt\") pod \"maas-api-revoke-sub-7639d7e8-zwlsr\" (UID: \"07d5d4c6-25ee-4574-b508-7c9ef5b3be45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7639d7e8-zwlsr"
Nov 03 08:56:08.559960 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:08.559925    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7639d7e8-zwlsr"
Nov 03 08:56:08.817379 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:08.817312    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 08:56:08.934240 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:08.934212    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7639d7e8-zwlsr"]
Nov 03 08:56:08.936258 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:56:08.936230    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07d5d4c6_25ee_4574_b508_7c9ef5b3be45.slice/crio-9ed246b3922de08fdf8e6b2712429fc15b34f37f07ac6232b48f87276d04dbbb WatchSource:0}: Error finding container 9ed246b3922de08fdf8e6b2712429fc15b34f37f07ac6232b48f87276d04dbbb: Status 404 returned error can't find the container with id 9ed246b3922de08fdf8e6b2712429fc15b34f37f07ac6232b48f87276d04dbbb
Nov 03 08:56:09.413754 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:09.413665    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7639d7e8-zwlsr" event={"ID":"07d5d4c6-25ee-4574-b508-7c9ef5b3be45","Type":"ContainerStarted","Data":"9da526e550e9ec00029bc8df226fd49c32c5c5cd525249659a65c40b9035ad43"}
Nov 03 08:56:09.413754 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:09.413748    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7639d7e8-zwlsr" event={"ID":"07d5d4c6-25ee-4574-b508-7c9ef5b3be45","Type":"ContainerStarted","Data":"9ed246b3922de08fdf8e6b2712429fc15b34f37f07ac6232b48f87276d04dbbb"}
Nov 03 08:56:09.534676 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:09.534618    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7639d7e8-zwlsr" podStartSLOduration=1.534600395 podStartE2EDuration="1.534600395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:56:09.533695059 +0000 UTC m=+993.802758517" watchObservedRunningTime="2026-11-03 08:56:09.534600395 +0000 UTC m=+993.803663843"
Nov 03 08:56:10.420465 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:10.420432    4473 generic.go:356] "Generic (PLEG): container finished" podID="07d5d4c6-25ee-4574-b508-7c9ef5b3be45" containerID="9da526e550e9ec00029bc8df226fd49c32c5c5cd525249659a65c40b9035ad43" exitCode=0
Nov 03 08:56:10.420852 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:10.420490    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7639d7e8-zwlsr" event={"ID":"07d5d4c6-25ee-4574-b508-7c9ef5b3be45","Type":"ContainerDied","Data":"9da526e550e9ec00029bc8df226fd49c32c5c5cd525249659a65c40b9035ad43"}
Nov 03 08:56:11.560542 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:11.560519    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7639d7e8-zwlsr"
Nov 03 08:56:11.615522 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:11.615499    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07d5d4c6-25ee-4574-b508-7c9ef5b3be45-kube-api-access-bhdkt\" (UniqueName: \"kubernetes.io/projected/07d5d4c6-25ee-4574-b508-7c9ef5b3be45-kube-api-access-bhdkt\") pod \"07d5d4c6-25ee-4574-b508-7c9ef5b3be45\" (UID: \"07d5d4c6-25ee-4574-b508-7c9ef5b3be45\") "
Nov 03 08:56:11.615648 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:11.615537    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07d5d4c6-25ee-4574-b508-7c9ef5b3be45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07d5d4c6-25ee-4574-b508-7c9ef5b3be45-maas-api-openshift-ca\") pod \"07d5d4c6-25ee-4574-b508-7c9ef5b3be45\" (UID: \"07d5d4c6-25ee-4574-b508-7c9ef5b3be45\") "
Nov 03 08:56:11.615648 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:11.615640    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07d5d4c6-25ee-4574-b508-7c9ef5b3be45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07d5d4c6-25ee-4574-b508-7c9ef5b3be45-maas-api-xks-ca\") pod \"07d5d4c6-25ee-4574-b508-7c9ef5b3be45\" (UID: \"07d5d4c6-25ee-4574-b508-7c9ef5b3be45\") "
Nov 03 08:56:11.615991 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:11.615968    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07d5d4c6-25ee-4574-b508-7c9ef5b3be45-maas-api-openshift-ca" pod "07d5d4c6-25ee-4574-b508-7c9ef5b3be45" (UID: "07d5d4c6-25ee-4574-b508-7c9ef5b3be45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:56:11.617819 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:11.617798    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07d5d4c6-25ee-4574-b508-7c9ef5b3be45-kube-api-access-bhdkt" pod "07d5d4c6-25ee-4574-b508-7c9ef5b3be45" (UID: "07d5d4c6-25ee-4574-b508-7c9ef5b3be45"). InnerVolumeSpecName "kube-api-access-bhdkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:56:11.617911 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:11.617794    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07d5d4c6-25ee-4574-b508-7c9ef5b3be45-maas-api-xks-ca" pod "07d5d4c6-25ee-4574-b508-7c9ef5b3be45" (UID: "07d5d4c6-25ee-4574-b508-7c9ef5b3be45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:56:11.717070 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:11.716971    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhdkt\" (UniqueName: \"kubernetes.io/projected/07d5d4c6-25ee-4574-b508-7c9ef5b3be45-kube-api-access-bhdkt\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:56:11.717070 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:11.717001    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07d5d4c6-25ee-4574-b508-7c9ef5b3be45-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:56:11.717070 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:11.717011    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07d5d4c6-25ee-4574-b508-7c9ef5b3be45-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:56:12.433141 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:12.433112    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7639d7e8-zwlsr"
Nov 03 08:56:12.433321 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:12.433150    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7639d7e8-zwlsr" event={"ID":"07d5d4c6-25ee-4574-b508-7c9ef5b3be45","Type":"ContainerDied","Data":"9ed246b3922de08fdf8e6b2712429fc15b34f37f07ac6232b48f87276d04dbbb"}
Nov 03 08:56:12.433321 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:12.433207    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ed246b3922de08fdf8e6b2712429fc15b34f37f07ac6232b48f87276d04dbbb"
Nov 03 08:56:14.068146 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:14.068106    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6427e717-nvch9"]
Nov 03 08:56:14.068759 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:14.068740    4473 cpu_manager.go:414] "containerMap: removing container" podUID="07d5d4c6-25ee-4574-b508-7c9ef5b3be45" containerName="revoke-keys"
Nov 03 08:56:14.068851 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:14.068760    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07d5d4c6-25ee-4574-b508-7c9ef5b3be45" containerName="revoke-keys"
Nov 03 08:56:14.068931 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:14.068917    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="07d5d4c6-25ee-4574-b508-7c9ef5b3be45" containerName="revoke-keys"
Nov 03 08:56:14.072206 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:14.072186    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6427e717-nvch9"
Nov 03 08:56:14.137154 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:14.137126    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aa5fad9-a1bc-4e3a-8d09-226887ac86c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6427e717-nvch9\" (UID: \"1aa5fad9-a1bc-4e3a-8d09-226887ac86c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6427e717-nvch9"
Nov 03 08:56:14.137343 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:14.137206    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7qwh\" (UniqueName: \"kubernetes.io/projected/1aa5fad9-a1bc-4e3a-8d09-226887ac86c5-kube-api-access-g7qwh\") pod \"maas-api-revoke-sub-6427e717-nvch9\" (UID: \"1aa5fad9-a1bc-4e3a-8d09-226887ac86c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6427e717-nvch9"
Nov 03 08:56:14.137419 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:14.137393    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aa5fad9-a1bc-4e3a-8d09-226887ac86c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6427e717-nvch9\" (UID: \"1aa5fad9-a1bc-4e3a-8d09-226887ac86c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6427e717-nvch9"
Nov 03 08:56:14.177004 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:14.176977    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:56:14.177004 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:14.176983    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:56:14.177197 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:14.177025    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:56:14.177197 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:14.177088    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 08:56:14.179093 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:14.179072    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6427e717-nvch9"]
Nov 03 08:56:14.239029 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:14.238994    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aa5fad9-a1bc-4e3a-8d09-226887ac86c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6427e717-nvch9\" (UID: \"1aa5fad9-a1bc-4e3a-8d09-226887ac86c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6427e717-nvch9"
Nov 03 08:56:14.239351 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:14.239322    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7qwh\" (UniqueName: \"kubernetes.io/projected/1aa5fad9-a1bc-4e3a-8d09-226887ac86c5-kube-api-access-g7qwh\") pod \"maas-api-revoke-sub-6427e717-nvch9\" (UID: \"1aa5fad9-a1bc-4e3a-8d09-226887ac86c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6427e717-nvch9"
Nov 03 08:56:14.239613 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:14.239593    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aa5fad9-a1bc-4e3a-8d09-226887ac86c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6427e717-nvch9\" (UID: \"1aa5fad9-a1bc-4e3a-8d09-226887ac86c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6427e717-nvch9"
Nov 03 08:56:14.240585 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:14.240559    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aa5fad9-a1bc-4e3a-8d09-226887ac86c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6427e717-nvch9\" (UID: \"1aa5fad9-a1bc-4e3a-8d09-226887ac86c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6427e717-nvch9"
Nov 03 08:56:14.243977 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:14.243926    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aa5fad9-a1bc-4e3a-8d09-226887ac86c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6427e717-nvch9\" (UID: \"1aa5fad9-a1bc-4e3a-8d09-226887ac86c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6427e717-nvch9"
Nov 03 08:56:14.303359 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:14.303321    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7qwh\" (UniqueName: \"kubernetes.io/projected/1aa5fad9-a1bc-4e3a-8d09-226887ac86c5-kube-api-access-g7qwh\") pod \"maas-api-revoke-sub-6427e717-nvch9\" (UID: \"1aa5fad9-a1bc-4e3a-8d09-226887ac86c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6427e717-nvch9"
Nov 03 08:56:14.385241 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:14.385205    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6427e717-nvch9"
Nov 03 08:56:14.792422 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:14.792398    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6427e717-nvch9"]
Nov 03 08:56:14.794218 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:56:14.794188    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aa5fad9_a1bc_4e3a_8d09_226887ac86c5.slice/crio-b1fb838872946e4ad1b52ee38c92f13c77436133afd3ba8cbd9bb40f61b5340b WatchSource:0}: Error finding container b1fb838872946e4ad1b52ee38c92f13c77436133afd3ba8cbd9bb40f61b5340b: Status 404 returned error can't find the container with id b1fb838872946e4ad1b52ee38c92f13c77436133afd3ba8cbd9bb40f61b5340b
Nov 03 08:56:15.453864 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:15.453766    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6427e717-nvch9" event={"ID":"1aa5fad9-a1bc-4e3a-8d09-226887ac86c5","Type":"ContainerStarted","Data":"47f8b4007463815a7d201a2a538af86097763f6672699381a8f8ed835143019e"}
Nov 03 08:56:15.453864 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:15.453860    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6427e717-nvch9" event={"ID":"1aa5fad9-a1bc-4e3a-8d09-226887ac86c5","Type":"ContainerStarted","Data":"b1fb838872946e4ad1b52ee38c92f13c77436133afd3ba8cbd9bb40f61b5340b"}
Nov 03 08:56:15.572558 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:15.572508    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6427e717-nvch9" podStartSLOduration=1.572495046 podStartE2EDuration="1.572495046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:56:15.570705252 +0000 UTC m=+999.839768707" watchObservedRunningTime="2026-11-03 08:56:15.572495046 +0000 UTC m=+999.841558519"
Nov 03 08:56:16.459810 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:16.459777    4473 generic.go:356] "Generic (PLEG): container finished" podID="1aa5fad9-a1bc-4e3a-8d09-226887ac86c5" containerID="47f8b4007463815a7d201a2a538af86097763f6672699381a8f8ed835143019e" exitCode=0
Nov 03 08:56:16.460221 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:16.459865    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6427e717-nvch9" event={"ID":"1aa5fad9-a1bc-4e3a-8d09-226887ac86c5","Type":"ContainerDied","Data":"47f8b4007463815a7d201a2a538af86097763f6672699381a8f8ed835143019e"}
Nov 03 08:56:16.936686 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:56:16.936653    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:56:17.593024 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:17.593002    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6427e717-nvch9"
Nov 03 08:56:17.671968 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:17.671924    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aa5fad9-a1bc-4e3a-8d09-226887ac86c5-kube-api-access-g7qwh\" (UniqueName: \"kubernetes.io/projected/1aa5fad9-a1bc-4e3a-8d09-226887ac86c5-kube-api-access-g7qwh\") pod \"1aa5fad9-a1bc-4e3a-8d09-226887ac86c5\" (UID: \"1aa5fad9-a1bc-4e3a-8d09-226887ac86c5\") "
Nov 03 08:56:17.672085 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:17.672035    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1aa5fad9-a1bc-4e3a-8d09-226887ac86c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aa5fad9-a1bc-4e3a-8d09-226887ac86c5-maas-api-xks-ca\") pod \"1aa5fad9-a1bc-4e3a-8d09-226887ac86c5\" (UID: \"1aa5fad9-a1bc-4e3a-8d09-226887ac86c5\") "
Nov 03 08:56:17.672135 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:17.672103    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aa5fad9-a1bc-4e3a-8d09-226887ac86c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aa5fad9-a1bc-4e3a-8d09-226887ac86c5-maas-api-openshift-ca\") pod \"1aa5fad9-a1bc-4e3a-8d09-226887ac86c5\" (UID: \"1aa5fad9-a1bc-4e3a-8d09-226887ac86c5\") "
Nov 03 08:56:17.672442 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:17.672415    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aa5fad9-a1bc-4e3a-8d09-226887ac86c5-maas-api-openshift-ca" pod "1aa5fad9-a1bc-4e3a-8d09-226887ac86c5" (UID: "1aa5fad9-a1bc-4e3a-8d09-226887ac86c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:56:17.674199 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:17.674180    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aa5fad9-a1bc-4e3a-8d09-226887ac86c5-kube-api-access-g7qwh" pod "1aa5fad9-a1bc-4e3a-8d09-226887ac86c5" (UID: "1aa5fad9-a1bc-4e3a-8d09-226887ac86c5"). InnerVolumeSpecName "kube-api-access-g7qwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:56:17.674268 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:17.674224    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1aa5fad9-a1bc-4e3a-8d09-226887ac86c5-maas-api-xks-ca" pod "1aa5fad9-a1bc-4e3a-8d09-226887ac86c5" (UID: "1aa5fad9-a1bc-4e3a-8d09-226887ac86c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:56:17.773294 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:17.773225    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aa5fad9-a1bc-4e3a-8d09-226887ac86c5-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:56:17.773294 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:17.773249    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aa5fad9-a1bc-4e3a-8d09-226887ac86c5-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:56:17.773294 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:17.773260    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7qwh\" (UniqueName: \"kubernetes.io/projected/1aa5fad9-a1bc-4e3a-8d09-226887ac86c5-kube-api-access-g7qwh\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:56:18.470753 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:18.470716    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6427e717-nvch9" event={"ID":"1aa5fad9-a1bc-4e3a-8d09-226887ac86c5","Type":"ContainerDied","Data":"b1fb838872946e4ad1b52ee38c92f13c77436133afd3ba8cbd9bb40f61b5340b"}
Nov 03 08:56:18.470753 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:18.470751    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6427e717-nvch9"
Nov 03 08:56:18.471033 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:18.470775    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1fb838872946e4ad1b52ee38c92f13c77436133afd3ba8cbd9bb40f61b5340b"
Nov 03 08:56:23.811494 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:23.811455    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a3eba79-h9hvz"]
Nov 03 08:56:23.812082 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:23.812067    4473 cpu_manager.go:414] "containerMap: removing container" podUID="1aa5fad9-a1bc-4e3a-8d09-226887ac86c5" containerName="revoke-keys"
Nov 03 08:56:23.812137 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:23.812085    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aa5fad9-a1bc-4e3a-8d09-226887ac86c5" containerName="revoke-keys"
Nov 03 08:56:23.812201 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:23.812191    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aa5fad9-a1bc-4e3a-8d09-226887ac86c5" containerName="revoke-keys"
Nov 03 08:56:23.815652 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:23.815635    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3eba79-h9hvz"
Nov 03 08:56:23.873207 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:23.873179    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a3eba79-h9hvz"]
Nov 03 08:56:23.920339 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:23.920306    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:56:23.920339 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:23.920320    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:56:23.920339 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:23.920316    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:56:23.920563 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:23.920359    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 08:56:23.944099 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:23.944074    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f8cb3cb-6a02-463d-a8c3-cacbf5449af5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a3eba79-h9hvz\" (UID: \"3f8cb3cb-6a02-463d-a8c3-cacbf5449af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3eba79-h9hvz"
Nov 03 08:56:23.944220 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:23.944200    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svml9\" (UniqueName: \"kubernetes.io/projected/3f8cb3cb-6a02-463d-a8c3-cacbf5449af5-kube-api-access-svml9\") pod \"maas-api-revoke-sub-8a3eba79-h9hvz\" (UID: \"3f8cb3cb-6a02-463d-a8c3-cacbf5449af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3eba79-h9hvz"
Nov 03 08:56:23.944263 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:23.944253    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f8cb3cb-6a02-463d-a8c3-cacbf5449af5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a3eba79-h9hvz\" (UID: \"3f8cb3cb-6a02-463d-a8c3-cacbf5449af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3eba79-h9hvz"
Nov 03 08:56:24.045096 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:24.045070    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svml9\" (UniqueName: \"kubernetes.io/projected/3f8cb3cb-6a02-463d-a8c3-cacbf5449af5-kube-api-access-svml9\") pod \"maas-api-revoke-sub-8a3eba79-h9hvz\" (UID: \"3f8cb3cb-6a02-463d-a8c3-cacbf5449af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3eba79-h9hvz"
Nov 03 08:56:24.045260 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:24.045118    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f8cb3cb-6a02-463d-a8c3-cacbf5449af5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a3eba79-h9hvz\" (UID: \"3f8cb3cb-6a02-463d-a8c3-cacbf5449af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3eba79-h9hvz"
Nov 03 08:56:24.045323 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:24.045257    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f8cb3cb-6a02-463d-a8c3-cacbf5449af5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a3eba79-h9hvz\" (UID: \"3f8cb3cb-6a02-463d-a8c3-cacbf5449af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3eba79-h9hvz"
Nov 03 08:56:24.045981 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:24.045884    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f8cb3cb-6a02-463d-a8c3-cacbf5449af5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a3eba79-h9hvz\" (UID: \"3f8cb3cb-6a02-463d-a8c3-cacbf5449af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3eba79-h9hvz"
Nov 03 08:56:24.047578 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:24.047559    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f8cb3cb-6a02-463d-a8c3-cacbf5449af5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a3eba79-h9hvz\" (UID: \"3f8cb3cb-6a02-463d-a8c3-cacbf5449af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3eba79-h9hvz"
Nov 03 08:56:24.104672 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:24.104612    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svml9\" (UniqueName: \"kubernetes.io/projected/3f8cb3cb-6a02-463d-a8c3-cacbf5449af5-kube-api-access-svml9\") pod \"maas-api-revoke-sub-8a3eba79-h9hvz\" (UID: \"3f8cb3cb-6a02-463d-a8c3-cacbf5449af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3eba79-h9hvz"
Nov 03 08:56:24.128444 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:24.128423    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3eba79-h9hvz"
Nov 03 08:56:24.504237 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:24.504210    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a3eba79-h9hvz"]
Nov 03 08:56:24.506127 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:56:24.506098    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f8cb3cb_6a02_463d_a8c3_cacbf5449af5.slice/crio-2ae63135714caa06ffe96d2c2f41077f50aa332c65d7658bc3b126d1800ea890 WatchSource:0}: Error finding container 2ae63135714caa06ffe96d2c2f41077f50aa332c65d7658bc3b126d1800ea890: Status 404 returned error can't find the container with id 2ae63135714caa06ffe96d2c2f41077f50aa332c65d7658bc3b126d1800ea890
Nov 03 08:56:25.509691 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:25.509654    4473 generic.go:356] "Generic (PLEG): container finished" podID="3f8cb3cb-6a02-463d-a8c3-cacbf5449af5" containerID="50174e5684c2820cd1a902c14fa5b978f853c0b5b715ab7be9e6b50512c7b7c2" exitCode=0
Nov 03 08:56:25.510092 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:25.509700    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3eba79-h9hvz" event={"ID":"3f8cb3cb-6a02-463d-a8c3-cacbf5449af5","Type":"ContainerDied","Data":"50174e5684c2820cd1a902c14fa5b978f853c0b5b715ab7be9e6b50512c7b7c2"}
Nov 03 08:56:25.510092 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:25.509790    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3eba79-h9hvz" event={"ID":"3f8cb3cb-6a02-463d-a8c3-cacbf5449af5","Type":"ContainerStarted","Data":"2ae63135714caa06ffe96d2c2f41077f50aa332c65d7658bc3b126d1800ea890"}
Nov 03 08:56:26.647287 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:26.647262    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3eba79-h9hvz"
Nov 03 08:56:26.771558 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:26.771467    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f8cb3cb-6a02-463d-a8c3-cacbf5449af5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f8cb3cb-6a02-463d-a8c3-cacbf5449af5-maas-api-xks-ca\") pod \"3f8cb3cb-6a02-463d-a8c3-cacbf5449af5\" (UID: \"3f8cb3cb-6a02-463d-a8c3-cacbf5449af5\") "
Nov 03 08:56:26.771558 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:26.771547    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f8cb3cb-6a02-463d-a8c3-cacbf5449af5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f8cb3cb-6a02-463d-a8c3-cacbf5449af5-maas-api-openshift-ca\") pod \"3f8cb3cb-6a02-463d-a8c3-cacbf5449af5\" (UID: \"3f8cb3cb-6a02-463d-a8c3-cacbf5449af5\") "
Nov 03 08:56:26.771783 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:26.771681    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f8cb3cb-6a02-463d-a8c3-cacbf5449af5-kube-api-access-svml9\" (UniqueName: \"kubernetes.io/projected/3f8cb3cb-6a02-463d-a8c3-cacbf5449af5-kube-api-access-svml9\") pod \"3f8cb3cb-6a02-463d-a8c3-cacbf5449af5\" (UID: \"3f8cb3cb-6a02-463d-a8c3-cacbf5449af5\") "
Nov 03 08:56:26.772007 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:26.771977    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f8cb3cb-6a02-463d-a8c3-cacbf5449af5-maas-api-openshift-ca" pod "3f8cb3cb-6a02-463d-a8c3-cacbf5449af5" (UID: "3f8cb3cb-6a02-463d-a8c3-cacbf5449af5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:56:26.774259 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:26.774230    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f8cb3cb-6a02-463d-a8c3-cacbf5449af5-maas-api-xks-ca" pod "3f8cb3cb-6a02-463d-a8c3-cacbf5449af5" (UID: "3f8cb3cb-6a02-463d-a8c3-cacbf5449af5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:56:26.774613 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:26.774597    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f8cb3cb-6a02-463d-a8c3-cacbf5449af5-kube-api-access-svml9" pod "3f8cb3cb-6a02-463d-a8c3-cacbf5449af5" (UID: "3f8cb3cb-6a02-463d-a8c3-cacbf5449af5"). InnerVolumeSpecName "kube-api-access-svml9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:56:26.872833 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:26.872804    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f8cb3cb-6a02-463d-a8c3-cacbf5449af5-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:56:26.872833 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:26.872831    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f8cb3cb-6a02-463d-a8c3-cacbf5449af5-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:56:26.872833 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:26.872841    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svml9\" (UniqueName: \"kubernetes.io/projected/3f8cb3cb-6a02-463d-a8c3-cacbf5449af5-kube-api-access-svml9\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:56:27.520352 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:27.520321    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3eba79-h9hvz"
Nov 03 08:56:27.520521 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:27.520366    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3eba79-h9hvz" event={"ID":"3f8cb3cb-6a02-463d-a8c3-cacbf5449af5","Type":"ContainerDied","Data":"2ae63135714caa06ffe96d2c2f41077f50aa332c65d7658bc3b126d1800ea890"}
Nov 03 08:56:27.520521 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:27.520413    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ae63135714caa06ffe96d2c2f41077f50aa332c65d7658bc3b126d1800ea890"
Nov 03 08:56:33.322288 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:33.322250    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 08:56:44.143125 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:44.143087    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"]
Nov 03 08:56:44.143530 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:44.143505    4473 cpu_manager.go:414] "containerMap: removing container" podUID="3f8cb3cb-6a02-463d-a8c3-cacbf5449af5" containerName="revoke-keys"
Nov 03 08:56:44.143530 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:44.143516    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f8cb3cb-6a02-463d-a8c3-cacbf5449af5" containerName="revoke-keys"
Nov 03 08:56:44.143610 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:44.143587    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f8cb3cb-6a02-463d-a8c3-cacbf5449af5" containerName="revoke-keys"
Nov 03 08:56:44.147977 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:44.147938    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"
Nov 03 08:56:44.202433 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:44.202406    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-18f9da-openshift-default-dockercfg-zmkbm\""
Nov 03 08:56:44.228687 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:44.228658    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/165b0188-6916-4776-9059-7a4f765a13f0-istiod-ca-cert\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr\" (UID: \"165b0188-6916-4776-9059-7a4f765a13f0\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"
Nov 03 08:56:44.228837 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:44.228694    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/165b0188-6916-4776-9059-7a4f765a13f0-istio-podinfo\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr\" (UID: \"165b0188-6916-4776-9059-7a4f765a13f0\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"
Nov 03 08:56:44.228837 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:44.228715    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/165b0188-6916-4776-9059-7a4f765a13f0-istio-token\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr\" (UID: \"165b0188-6916-4776-9059-7a4f765a13f0\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"
Nov 03 08:56:44.228837 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:44.228821    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/165b0188-6916-4776-9059-7a4f765a13f0-istio-envoy\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr\" (UID: \"165b0188-6916-4776-9059-7a4f765a13f0\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"
Nov 03 08:56:44.228990 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:44.228871    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/165b0188-6916-4776-9059-7a4f765a13f0-workload-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr\" (UID: \"165b0188-6916-4776-9059-7a4f765a13f0\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"
Nov 03 08:56:44.228990 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:44.228970    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/165b0188-6916-4776-9059-7a4f765a13f0-istio-data\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr\" (UID: \"165b0188-6916-4776-9059-7a4f765a13f0\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"
Nov 03 08:56:44.229066 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:44.229017    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jbll\" (UniqueName: \"kubernetes.io/projected/165b0188-6916-4776-9059-7a4f765a13f0-kube-api-access-6jbll\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr\" (UID: \"165b0188-6916-4776-9059-7a4f765a13f0\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"
Nov 03 08:56:44.229110 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:44.229075    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/165b0188-6916-4776-9059-7a4f765a13f0-workload-certs\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr\" (UID: \"165b0188-6916-4776-9059-7a4f765a13f0\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"
Nov 03 08:56:44.229110 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:44.229102    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/165b0188-6916-4776-9059-7a4f765a13f0-credential-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr\" (UID: \"165b0188-6916-4776-9059-7a4f765a13f0\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"
Nov 03 08:56:44.255748 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:44.255718    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"]
Nov 03 08:56:44.329897 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:44.329870    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/165b0188-6916-4776-9059-7a4f765a13f0-istiod-ca-cert\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr\" (UID: \"165b0188-6916-4776-9059-7a4f765a13f0\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"
Nov 03 08:56:44.330080 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:44.329904    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/165b0188-6916-4776-9059-7a4f765a13f0-istio-podinfo\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr\" (UID: \"165b0188-6916-4776-9059-7a4f765a13f0\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"
Nov 03 08:56:44.330080 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:44.330024    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/165b0188-6916-4776-9059-7a4f765a13f0-istio-token\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr\" (UID: \"165b0188-6916-4776-9059-7a4f765a13f0\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"
Nov 03 08:56:44.330080 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:44.330064    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/165b0188-6916-4776-9059-7a4f765a13f0-istio-envoy\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr\" (UID: \"165b0188-6916-4776-9059-7a4f765a13f0\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"
Nov 03 08:56:44.330225 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:44.330112    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/165b0188-6916-4776-9059-7a4f765a13f0-workload-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr\" (UID: \"165b0188-6916-4776-9059-7a4f765a13f0\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"
Nov 03 08:56:44.330225 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:44.330178    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/165b0188-6916-4776-9059-7a4f765a13f0-istio-data\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr\" (UID: \"165b0188-6916-4776-9059-7a4f765a13f0\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"
Nov 03 08:56:44.330316 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:44.330233    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jbll\" (UniqueName: \"kubernetes.io/projected/165b0188-6916-4776-9059-7a4f765a13f0-kube-api-access-6jbll\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr\" (UID: \"165b0188-6916-4776-9059-7a4f765a13f0\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"
Nov 03 08:56:44.330316 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:44.330274    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/165b0188-6916-4776-9059-7a4f765a13f0-workload-certs\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr\" (UID: \"165b0188-6916-4776-9059-7a4f765a13f0\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"
Nov 03 08:56:44.330416 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:44.330315    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/165b0188-6916-4776-9059-7a4f765a13f0-credential-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr\" (UID: \"165b0188-6916-4776-9059-7a4f765a13f0\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"
Nov 03 08:56:44.330546 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:44.330519    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/165b0188-6916-4776-9059-7a4f765a13f0-workload-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr\" (UID: \"165b0188-6916-4776-9059-7a4f765a13f0\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"
Nov 03 08:56:44.330615 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:44.330584    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/165b0188-6916-4776-9059-7a4f765a13f0-istio-data\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr\" (UID: \"165b0188-6916-4776-9059-7a4f765a13f0\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"
Nov 03 08:56:44.330709 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:44.330681    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/165b0188-6916-4776-9059-7a4f765a13f0-credential-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr\" (UID: \"165b0188-6916-4776-9059-7a4f765a13f0\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"
Nov 03 08:56:44.330709 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:44.330696    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/165b0188-6916-4776-9059-7a4f765a13f0-workload-certs\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr\" (UID: \"165b0188-6916-4776-9059-7a4f765a13f0\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"
Nov 03 08:56:44.330852 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:44.330730    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/165b0188-6916-4776-9059-7a4f765a13f0-istiod-ca-cert\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr\" (UID: \"165b0188-6916-4776-9059-7a4f765a13f0\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"
Nov 03 08:56:44.332696 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:44.332665    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/165b0188-6916-4776-9059-7a4f765a13f0-istio-podinfo\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr\" (UID: \"165b0188-6916-4776-9059-7a4f765a13f0\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"
Nov 03 08:56:44.332809 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:44.332789    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/165b0188-6916-4776-9059-7a4f765a13f0-istio-envoy\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr\" (UID: \"165b0188-6916-4776-9059-7a4f765a13f0\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"
Nov 03 08:56:44.391104 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:44.391072    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jbll\" (UniqueName: \"kubernetes.io/projected/165b0188-6916-4776-9059-7a4f765a13f0-kube-api-access-6jbll\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr\" (UID: \"165b0188-6916-4776-9059-7a4f765a13f0\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"
Nov 03 08:56:44.391258 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:44.391114    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/165b0188-6916-4776-9059-7a4f765a13f0-istio-token\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr\" (UID: \"165b0188-6916-4776-9059-7a4f765a13f0\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"
Nov 03 08:56:44.460659 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:44.460582    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"
Nov 03 08:56:44.891853 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:44.891823    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"]
Nov 03 08:56:44.893428 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:56:44.893401    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod165b0188_6916_4776_9059_7a4f765a13f0.slice/crio-ae0ec75c5c110df1fd10f8369e53e6dd281f057d14cf8857814f9ba81d21f5c3 WatchSource:0}: Error finding container ae0ec75c5c110df1fd10f8369e53e6dd281f057d14cf8857814f9ba81d21f5c3: Status 404 returned error can't find the container with id ae0ec75c5c110df1fd10f8369e53e6dd281f057d14cf8857814f9ba81d21f5c3
Nov 03 08:56:44.895712 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:44.895681    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 08:56:44.895792 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:44.895745    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 08:56:44.895830 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:44.895793    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 08:56:45.647286 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:45.647247    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr" event={"ID":"165b0188-6916-4776-9059-7a4f765a13f0","Type":"ContainerStarted","Data":"ceac0af690d74ef235767727b311264e38c0c40bdb70a4b3217f0427856d585b"}
Nov 03 08:56:45.647714 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:45.647348    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr" event={"ID":"165b0188-6916-4776-9059-7a4f765a13f0","Type":"ContainerStarted","Data":"ae0ec75c5c110df1fd10f8369e53e6dd281f057d14cf8857814f9ba81d21f5c3"}
Nov 03 08:56:45.866165 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:45.866118    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr" podStartSLOduration=1.866105216 podStartE2EDuration="1.866105216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:56:45.815422998 +0000 UTC m=+1030.084486451" watchObservedRunningTime="2026-11-03 08:56:45.866105216 +0000 UTC m=+1030.135168649"
Nov 03 08:56:46.461120 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:46.461082    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"
Nov 03 08:56:46.462433 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:46.462401    4473 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 03 08:56:46.462571 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:46.462459    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr" podUID="165b0188-6916-4776-9059-7a4f765a13f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 03 08:56:46.961797 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:56:46.961719    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:56:47.461124 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:47.461089    4473 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 03 08:56:47.461304 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:47.461153    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr" podUID="165b0188-6916-4776-9059-7a4f765a13f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 03 08:56:48.462234 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:48.462196    4473 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 03 08:56:48.462718 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:48.462267    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr" podUID="165b0188-6916-4776-9059-7a4f765a13f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 03 08:56:49.461793 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:49.461341    4473 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 03 08:56:49.461793 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:49.461412    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr" podUID="165b0188-6916-4776-9059-7a4f765a13f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 03 08:56:50.461587 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:50.461545    4473 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 03 08:56:50.462277 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:50.462217    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr" podUID="165b0188-6916-4776-9059-7a4f765a13f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 03 08:56:51.789302 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:51.789262    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"
Nov 03 08:56:51.789752 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:51.789684    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"
Nov 03 08:56:51.790568 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:56:51.790548    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"
Nov 03 08:57:03.969496 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:03.969464    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 08:57:16.936731 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:57:16.936679    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:57:18.684481 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:18.684444    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"]
Nov 03 08:57:18.688367 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:18.688344    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"
Nov 03 08:57:18.841000 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:18.840932    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8m98b\" (UID: \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"
Nov 03 08:57:18.841221 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:18.841044    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8m98b\" (UID: \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"
Nov 03 08:57:18.841221 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:18.841113    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8m98b\" (UID: \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"
Nov 03 08:57:18.841354 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:18.841223    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8m98b\" (UID: \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"
Nov 03 08:57:18.841354 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:18.841250    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm5jz\" (UniqueName: \"kubernetes.io/projected/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-kube-api-access-dm5jz\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8m98b\" (UID: \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"
Nov 03 08:57:18.841354 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:18.841300    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8m98b\" (UID: \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"
Nov 03 08:57:18.841354 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:18.841325    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8m98b\" (UID: \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"
Nov 03 08:57:18.841354 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:18.841349    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8m98b\" (UID: \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"
Nov 03 08:57:18.841604 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:18.841382    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8m98b\" (UID: \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"
Nov 03 08:57:18.927403 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:18.927370    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"]
Nov 03 08:57:18.942456 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:18.942358    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm5jz\" (UniqueName: \"kubernetes.io/projected/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-kube-api-access-dm5jz\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8m98b\" (UID: \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"
Nov 03 08:57:18.943267 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:18.943239    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8m98b\" (UID: \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"
Nov 03 08:57:18.946252 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:18.946227    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8m98b\" (UID: \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"
Nov 03 08:57:18.946402 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:18.946262    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8m98b\" (UID: \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"
Nov 03 08:57:18.946402 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:18.946168    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8m98b\" (UID: \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"
Nov 03 08:57:18.946402 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:18.946304    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8m98b\" (UID: \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"
Nov 03 08:57:18.946402 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:18.946388    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8m98b\" (UID: \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"
Nov 03 08:57:18.946618 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:18.946464    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8m98b\" (UID: \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"
Nov 03 08:57:18.946618 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:18.946516    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8m98b\" (UID: \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"
Nov 03 08:57:18.946716 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:18.946652    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8m98b\" (UID: \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"
Nov 03 08:57:18.946842 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:18.946818    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8m98b\" (UID: \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"
Nov 03 08:57:18.947236 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:18.947213    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8m98b\" (UID: \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"
Nov 03 08:57:18.947575 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:18.947514    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8m98b\" (UID: \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"
Nov 03 08:57:18.948401 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:18.948378    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8m98b\" (UID: \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"
Nov 03 08:57:18.950016 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:18.949993    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8m98b\" (UID: \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"
Nov 03 08:57:18.951566 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:18.951525    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8m98b\" (UID: \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"
Nov 03 08:57:19.002640 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:19.002600    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm5jz\" (UniqueName: \"kubernetes.io/projected/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-kube-api-access-dm5jz\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8m98b\" (UID: \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"
Nov 03 08:57:19.005835 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:19.005803    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8m98b\" (UID: \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"
Nov 03 08:57:19.302104 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:19.302025    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"
Nov 03 08:57:19.747267 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:19.747239    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"]
Nov 03 08:57:19.748568 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:57:19.748531    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9252f6a_e0b5_45e4_8625_cc1af53d61c2.slice/crio-f899c9086f44388ac20ac2ff7bafb221b40a6ffd667fbd1dd93378e67c7d6c82 WatchSource:0}: Error finding container f899c9086f44388ac20ac2ff7bafb221b40a6ffd667fbd1dd93378e67c7d6c82: Status 404 returned error can't find the container with id f899c9086f44388ac20ac2ff7bafb221b40a6ffd667fbd1dd93378e67c7d6c82
Nov 03 08:57:19.750871 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:19.750836    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 08:57:19.751000 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:19.750932    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 08:57:19.751074 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:19.750999    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 08:57:19.874041 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:19.874011    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b" event={"ID":"e9252f6a-e0b5-45e4-8625-cc1af53d61c2","Type":"ContainerStarted","Data":"59e5d9ef838089cb0443567129743b58489774ce2b452573c2b781fc13848a82"}
Nov 03 08:57:19.874198 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:19.874057    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b" event={"ID":"e9252f6a-e0b5-45e4-8625-cc1af53d61c2","Type":"ContainerStarted","Data":"f899c9086f44388ac20ac2ff7bafb221b40a6ffd667fbd1dd93378e67c7d6c82"}
Nov 03 08:57:20.063036 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:20.060236    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b" podStartSLOduration=2.060212571 podStartE2EDuration="2.060212571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:57:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:57:20.055009391 +0000 UTC m=+1064.324072848" watchObservedRunningTime="2026-11-03 08:57:20.060212571 +0000 UTC m=+1064.329276007"
Nov 03 08:57:20.303030 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:20.302995    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"
Nov 03 08:57:20.304228 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:20.304204    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8m98b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 08:57:20.304350 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:20.304257    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b" podUID="e9252f6a-e0b5-45e4-8625-cc1af53d61c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 08:57:21.302916 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:21.302882    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8m98b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 08:57:21.303352 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:21.302967    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b" podUID="e9252f6a-e0b5-45e4-8625-cc1af53d61c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 08:57:21.504758 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:21.504712    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-910d50/e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j"]
Nov 03 08:57:21.508477 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:21.508456    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j"
Nov 03 08:57:21.588295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:21.588214    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f1584e1-c962-4d28-8c23-7885eccc45f6-tmp-dir\") pod \"e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j\" (UID: \"0f1584e1-c962-4d28-8c23-7885eccc45f6\") " pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j"
Nov 03 08:57:21.588295 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:21.588257    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f1584e1-c962-4d28-8c23-7885eccc45f6-home\") pod \"e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j\" (UID: \"0f1584e1-c962-4d28-8c23-7885eccc45f6\") " pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j"
Nov 03 08:57:21.588478 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:21.588356    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f1584e1-c962-4d28-8c23-7885eccc45f6-model-cache\") pod \"e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j\" (UID: \"0f1584e1-c962-4d28-8c23-7885eccc45f6\") " pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j"
Nov 03 08:57:21.588478 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:21.588378    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9d7v\" (UniqueName: \"kubernetes.io/projected/0f1584e1-c962-4d28-8c23-7885eccc45f6-kube-api-access-k9d7v\") pod \"e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j\" (UID: \"0f1584e1-c962-4d28-8c23-7885eccc45f6\") " pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j"
Nov 03 08:57:21.588478 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:21.588460    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f1584e1-c962-4d28-8c23-7885eccc45f6-dshm\") pod \"e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j\" (UID: \"0f1584e1-c962-4d28-8c23-7885eccc45f6\") " pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j"
Nov 03 08:57:21.588590 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:21.588485    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f1584e1-c962-4d28-8c23-7885eccc45f6-tls-certs\") pod \"e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j\" (UID: \"0f1584e1-c962-4d28-8c23-7885eccc45f6\") " pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j"
Nov 03 08:57:21.614304 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:21.614276    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-910d50\"/\"kube-root-ca.crt\""
Nov 03 08:57:21.614453 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:21.614283    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-910d50\"/\"e2e-conflict-a-54a1bbf9-kserve-self-signed-certs\""
Nov 03 08:57:21.614453 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:21.614300    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-910d50\"/\"openshift-service-ca.crt\""
Nov 03 08:57:21.614453 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:21.614362    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-910d50\"/\"default-dockercfg-2bds8\""
Nov 03 08:57:21.629968 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:21.616818    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-910d50/e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j"]
Nov 03 08:57:21.689926 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:21.689896    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f1584e1-c962-4d28-8c23-7885eccc45f6-model-cache\") pod \"e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j\" (UID: \"0f1584e1-c962-4d28-8c23-7885eccc45f6\") " pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j"
Nov 03 08:57:21.690140 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:21.689933    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9d7v\" (UniqueName: \"kubernetes.io/projected/0f1584e1-c962-4d28-8c23-7885eccc45f6-kube-api-access-k9d7v\") pod \"e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j\" (UID: \"0f1584e1-c962-4d28-8c23-7885eccc45f6\") " pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j"
Nov 03 08:57:21.690140 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:21.689990    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f1584e1-c962-4d28-8c23-7885eccc45f6-dshm\") pod \"e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j\" (UID: \"0f1584e1-c962-4d28-8c23-7885eccc45f6\") " pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j"
Nov 03 08:57:21.690140 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:21.690008    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f1584e1-c962-4d28-8c23-7885eccc45f6-tls-certs\") pod \"e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j\" (UID: \"0f1584e1-c962-4d28-8c23-7885eccc45f6\") " pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j"
Nov 03 08:57:21.690140 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:21.690094    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f1584e1-c962-4d28-8c23-7885eccc45f6-tmp-dir\") pod \"e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j\" (UID: \"0f1584e1-c962-4d28-8c23-7885eccc45f6\") " pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j"
Nov 03 08:57:21.690140 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:21.690132    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f1584e1-c962-4d28-8c23-7885eccc45f6-home\") pod \"e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j\" (UID: \"0f1584e1-c962-4d28-8c23-7885eccc45f6\") " pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j"
Nov 03 08:57:21.690409 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:21.690302    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f1584e1-c962-4d28-8c23-7885eccc45f6-model-cache\") pod \"e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j\" (UID: \"0f1584e1-c962-4d28-8c23-7885eccc45f6\") " pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j"
Nov 03 08:57:21.690493 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:21.690475    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f1584e1-c962-4d28-8c23-7885eccc45f6-home\") pod \"e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j\" (UID: \"0f1584e1-c962-4d28-8c23-7885eccc45f6\") " pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j"
Nov 03 08:57:21.690554 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:21.690505    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f1584e1-c962-4d28-8c23-7885eccc45f6-tmp-dir\") pod \"e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j\" (UID: \"0f1584e1-c962-4d28-8c23-7885eccc45f6\") " pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j"
Nov 03 08:57:21.692375 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:21.692353    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f1584e1-c962-4d28-8c23-7885eccc45f6-dshm\") pod \"e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j\" (UID: \"0f1584e1-c962-4d28-8c23-7885eccc45f6\") " pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j"
Nov 03 08:57:21.693457 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:21.693442    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f1584e1-c962-4d28-8c23-7885eccc45f6-tls-certs\") pod \"e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j\" (UID: \"0f1584e1-c962-4d28-8c23-7885eccc45f6\") " pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j"
Nov 03 08:57:21.749827 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:21.749791    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9d7v\" (UniqueName: \"kubernetes.io/projected/0f1584e1-c962-4d28-8c23-7885eccc45f6-kube-api-access-k9d7v\") pod \"e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j\" (UID: \"0f1584e1-c962-4d28-8c23-7885eccc45f6\") " pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j"
Nov 03 08:57:21.851203 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:21.851109    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j"
Nov 03 08:57:22.244929 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:22.244900    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-910d50/e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j"]
Nov 03 08:57:22.246068 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:57:22.246021    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f1584e1_c962_4d28_8c23_7885eccc45f6.slice/crio-ad7bd2e66e2257f4bb227a97ae8ae81567b82e99e3f9d0dd331f6171a69a2dfd WatchSource:0}: Error finding container ad7bd2e66e2257f4bb227a97ae8ae81567b82e99e3f9d0dd331f6171a69a2dfd: Status 404 returned error can't find the container with id ad7bd2e66e2257f4bb227a97ae8ae81567b82e99e3f9d0dd331f6171a69a2dfd
Nov 03 08:57:22.302758 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:22.302737    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8m98b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 08:57:22.302865 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:22.302777    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b" podUID="e9252f6a-e0b5-45e4-8625-cc1af53d61c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 08:57:22.893715 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:22.893678    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j" event={"ID":"0f1584e1-c962-4d28-8c23-7885eccc45f6","Type":"ContainerStarted","Data":"4e2536c542b85ca9b67e7f27b5d47010600c2ede499eb0447ae78ca875ffae31"}
Nov 03 08:57:22.894207 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:22.893731    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j" event={"ID":"0f1584e1-c962-4d28-8c23-7885eccc45f6","Type":"ContainerStarted","Data":"ad7bd2e66e2257f4bb227a97ae8ae81567b82e99e3f9d0dd331f6171a69a2dfd"}
Nov 03 08:57:23.010893 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:23.010840    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j" podStartSLOduration=2.010824816 podStartE2EDuration="2.010824816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:57:23.009714847 +0000 UTC m=+1067.278778294" watchObservedRunningTime="2026-11-03 08:57:23.010824816 +0000 UTC m=+1067.279888250"
Nov 03 08:57:23.302925 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:23.302844    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8m98b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 08:57:23.302925 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:23.302899    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b" podUID="e9252f6a-e0b5-45e4-8625-cc1af53d61c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 08:57:23.852020 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:23.851923    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j"
Nov 03 08:57:24.303472 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:24.303431    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8m98b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 08:57:24.303996 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:24.303491    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b" podUID="e9252f6a-e0b5-45e4-8625-cc1af53d61c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 08:57:25.302786 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:25.302750    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8m98b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 08:57:25.302994 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:25.302813    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b" podUID="e9252f6a-e0b5-45e4-8625-cc1af53d61c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 08:57:26.303181 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:26.303149    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8m98b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 08:57:26.303598 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:26.303198    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b" podUID="e9252f6a-e0b5-45e4-8625-cc1af53d61c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 08:57:27.273079 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:27.273039    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 08:57:27.303312 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:27.303283    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8m98b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 08:57:27.303685 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:27.303332    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b" podUID="e9252f6a-e0b5-45e4-8625-cc1af53d61c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 08:57:28.914035 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:28.914009    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"
Nov 03 08:57:28.914923 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:28.914898    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"
Nov 03 08:57:28.916365 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:28.916345    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"
Nov 03 08:57:29.272725 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:29.272623    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34957a84-mpjdq"]
Nov 03 08:57:29.277262 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:29.277237    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34957a84-mpjdq"
Nov 03 08:57:29.333182 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:29.332890    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:57:29.333182 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:29.333031    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:57:29.333182 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:29.333115    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 08:57:29.333182 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:29.333122    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:57:29.333895 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:29.333871    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34957a84-mpjdq"]
Nov 03 08:57:29.427569 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:29.427533    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9c34e5b-23a9-4978-998a-26125e56212e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34957a84-mpjdq\" (UID: \"b9c34e5b-23a9-4978-998a-26125e56212e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34957a84-mpjdq"
Nov 03 08:57:29.427767 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:29.427745    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvz6h\" (UniqueName: \"kubernetes.io/projected/b9c34e5b-23a9-4978-998a-26125e56212e-kube-api-access-nvz6h\") pod \"maas-api-revoke-sub-34957a84-mpjdq\" (UID: \"b9c34e5b-23a9-4978-998a-26125e56212e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34957a84-mpjdq"
Nov 03 08:57:29.427832 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:29.427815    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9c34e5b-23a9-4978-998a-26125e56212e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34957a84-mpjdq\" (UID: \"b9c34e5b-23a9-4978-998a-26125e56212e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34957a84-mpjdq"
Nov 03 08:57:29.529567 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:29.528602    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9c34e5b-23a9-4978-998a-26125e56212e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34957a84-mpjdq\" (UID: \"b9c34e5b-23a9-4978-998a-26125e56212e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34957a84-mpjdq"
Nov 03 08:57:29.529567 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:29.528669    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9c34e5b-23a9-4978-998a-26125e56212e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34957a84-mpjdq\" (UID: \"b9c34e5b-23a9-4978-998a-26125e56212e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34957a84-mpjdq"
Nov 03 08:57:29.529567 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:29.528856    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvz6h\" (UniqueName: \"kubernetes.io/projected/b9c34e5b-23a9-4978-998a-26125e56212e-kube-api-access-nvz6h\") pod \"maas-api-revoke-sub-34957a84-mpjdq\" (UID: \"b9c34e5b-23a9-4978-998a-26125e56212e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34957a84-mpjdq"
Nov 03 08:57:29.529567 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:29.529413    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9c34e5b-23a9-4978-998a-26125e56212e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34957a84-mpjdq\" (UID: \"b9c34e5b-23a9-4978-998a-26125e56212e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34957a84-mpjdq"
Nov 03 08:57:29.551047 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:29.551015    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9c34e5b-23a9-4978-998a-26125e56212e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34957a84-mpjdq\" (UID: \"b9c34e5b-23a9-4978-998a-26125e56212e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34957a84-mpjdq"
Nov 03 08:57:29.602267 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:29.599237    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvz6h\" (UniqueName: \"kubernetes.io/projected/b9c34e5b-23a9-4978-998a-26125e56212e-kube-api-access-nvz6h\") pod \"maas-api-revoke-sub-34957a84-mpjdq\" (UID: \"b9c34e5b-23a9-4978-998a-26125e56212e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34957a84-mpjdq"
Nov 03 08:57:29.896837 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:29.896799    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34957a84-mpjdq"
Nov 03 08:57:30.394260 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:30.393772    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34957a84-mpjdq"]
Nov 03 08:57:30.394726 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:57:30.394413    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9c34e5b_23a9_4978_998a_26125e56212e.slice/crio-d0cbd0308f0da71a78e751a84064383faa5af4214d8d1a157595b7e9d176dc05 WatchSource:0}: Error finding container d0cbd0308f0da71a78e751a84064383faa5af4214d8d1a157595b7e9d176dc05: Status 404 returned error can't find the container with id d0cbd0308f0da71a78e751a84064383faa5af4214d8d1a157595b7e9d176dc05
Nov 03 08:57:31.000426 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:31.000332    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34957a84-mpjdq" event={"ID":"b9c34e5b-23a9-4978-998a-26125e56212e","Type":"ContainerStarted","Data":"96caadeb61d84012f7504067985c1e8053513f31f7017ec06494501abfebe9e6"}
Nov 03 08:57:31.000426 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:31.000388    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34957a84-mpjdq" event={"ID":"b9c34e5b-23a9-4978-998a-26125e56212e","Type":"ContainerStarted","Data":"d0cbd0308f0da71a78e751a84064383faa5af4214d8d1a157595b7e9d176dc05"}
Nov 03 08:57:31.142971 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:31.136037    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34957a84-mpjdq" podStartSLOduration=2.136015822 podStartE2EDuration="2.136015822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:57:31.121655445 +0000 UTC m=+1075.390718907" watchObservedRunningTime="2026-11-03 08:57:31.136015822 +0000 UTC m=+1075.405079257"
Nov 03 08:57:31.851692 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:31.851616    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j"
Nov 03 08:57:31.870170 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:31.870146    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j"
Nov 03 08:57:32.007682 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:32.007652    4473 generic.go:356] "Generic (PLEG): container finished" podID="b9c34e5b-23a9-4978-998a-26125e56212e" containerID="96caadeb61d84012f7504067985c1e8053513f31f7017ec06494501abfebe9e6" exitCode=0
Nov 03 08:57:32.007839 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:32.007736    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34957a84-mpjdq" event={"ID":"b9c34e5b-23a9-4978-998a-26125e56212e","Type":"ContainerDied","Data":"96caadeb61d84012f7504067985c1e8053513f31f7017ec06494501abfebe9e6"}
Nov 03 08:57:32.019141 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:32.019120    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j"
Nov 03 08:57:33.156305 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:33.156283    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34957a84-mpjdq"
Nov 03 08:57:33.221704 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:33.221677    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9c34e5b-23a9-4978-998a-26125e56212e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9c34e5b-23a9-4978-998a-26125e56212e-maas-api-xks-ca\") pod \"b9c34e5b-23a9-4978-998a-26125e56212e\" (UID: \"b9c34e5b-23a9-4978-998a-26125e56212e\") "
Nov 03 08:57:33.221834 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:33.221799    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9c34e5b-23a9-4978-998a-26125e56212e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9c34e5b-23a9-4978-998a-26125e56212e-maas-api-openshift-ca\") pod \"b9c34e5b-23a9-4978-998a-26125e56212e\" (UID: \"b9c34e5b-23a9-4978-998a-26125e56212e\") "
Nov 03 08:57:33.221834 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:33.221831    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9c34e5b-23a9-4978-998a-26125e56212e-kube-api-access-nvz6h\" (UniqueName: \"kubernetes.io/projected/b9c34e5b-23a9-4978-998a-26125e56212e-kube-api-access-nvz6h\") pod \"b9c34e5b-23a9-4978-998a-26125e56212e\" (UID: \"b9c34e5b-23a9-4978-998a-26125e56212e\") "
Nov 03 08:57:33.222211 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:33.222182    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9c34e5b-23a9-4978-998a-26125e56212e-maas-api-openshift-ca" pod "b9c34e5b-23a9-4978-998a-26125e56212e" (UID: "b9c34e5b-23a9-4978-998a-26125e56212e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:57:33.223886 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:33.223863    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9c34e5b-23a9-4978-998a-26125e56212e-maas-api-xks-ca" pod "b9c34e5b-23a9-4978-998a-26125e56212e" (UID: "b9c34e5b-23a9-4978-998a-26125e56212e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:57:33.224051 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:33.224026    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9c34e5b-23a9-4978-998a-26125e56212e-kube-api-access-nvz6h" pod "b9c34e5b-23a9-4978-998a-26125e56212e" (UID: "b9c34e5b-23a9-4978-998a-26125e56212e"). InnerVolumeSpecName "kube-api-access-nvz6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:57:33.322901 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:33.322835    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9c34e5b-23a9-4978-998a-26125e56212e-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:57:33.322901 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:33.322859    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvz6h\" (UniqueName: \"kubernetes.io/projected/b9c34e5b-23a9-4978-998a-26125e56212e-kube-api-access-nvz6h\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:57:33.322901 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:33.322869    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9c34e5b-23a9-4978-998a-26125e56212e-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 08:57:34.022698 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:34.022665    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34957a84-mpjdq"
Nov 03 08:57:34.022698 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:34.022694    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34957a84-mpjdq" event={"ID":"b9c34e5b-23a9-4978-998a-26125e56212e","Type":"ContainerDied","Data":"d0cbd0308f0da71a78e751a84064383faa5af4214d8d1a157595b7e9d176dc05"}
Nov 03 08:57:34.022925 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:34.022745    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0cbd0308f0da71a78e751a84064383faa5af4214d8d1a157595b7e9d176dc05"
Nov 03 08:57:44.949816 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:44.949776    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"]
Nov 03 08:57:44.950533 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:44.950513    4473 cpu_manager.go:414] "containerMap: removing container" podUID="b9c34e5b-23a9-4978-998a-26125e56212e" containerName="revoke-keys"
Nov 03 08:57:44.950586 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:44.950538    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9c34e5b-23a9-4978-998a-26125e56212e" containerName="revoke-keys"
Nov 03 08:57:44.950651 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:44.950639    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9c34e5b-23a9-4978-998a-26125e56212e" containerName="revoke-keys"
Nov 03 08:57:44.954460 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:44.954440    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"
Nov 03 08:57:45.008848 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:45.008820    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-910d50-openshift-default-dockercfg-w8xwd\""
Nov 03 08:57:45.047191 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:45.047159    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39799a31-d7b7-4948-af12-23cb1f426fc5-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th\" (UID: \"39799a31-d7b7-4948-af12-23cb1f426fc5\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"
Nov 03 08:57:45.047362 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:45.047242    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94z2p\" (UniqueName: \"kubernetes.io/projected/39799a31-d7b7-4948-af12-23cb1f426fc5-kube-api-access-94z2p\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th\" (UID: \"39799a31-d7b7-4948-af12-23cb1f426fc5\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"
Nov 03 08:57:45.047362 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:45.047341    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39799a31-d7b7-4948-af12-23cb1f426fc5-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th\" (UID: \"39799a31-d7b7-4948-af12-23cb1f426fc5\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"
Nov 03 08:57:45.047453 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:45.047380    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39799a31-d7b7-4948-af12-23cb1f426fc5-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th\" (UID: \"39799a31-d7b7-4948-af12-23cb1f426fc5\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"
Nov 03 08:57:45.047453 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:45.047418    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39799a31-d7b7-4948-af12-23cb1f426fc5-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th\" (UID: \"39799a31-d7b7-4948-af12-23cb1f426fc5\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"
Nov 03 08:57:45.047583 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:45.047555    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39799a31-d7b7-4948-af12-23cb1f426fc5-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th\" (UID: \"39799a31-d7b7-4948-af12-23cb1f426fc5\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"
Nov 03 08:57:45.047675 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:45.047659    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39799a31-d7b7-4948-af12-23cb1f426fc5-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th\" (UID: \"39799a31-d7b7-4948-af12-23cb1f426fc5\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"
Nov 03 08:57:45.047743 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:45.047716    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39799a31-d7b7-4948-af12-23cb1f426fc5-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th\" (UID: \"39799a31-d7b7-4948-af12-23cb1f426fc5\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"
Nov 03 08:57:45.047841 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:45.047787    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39799a31-d7b7-4948-af12-23cb1f426fc5-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th\" (UID: \"39799a31-d7b7-4948-af12-23cb1f426fc5\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"
Nov 03 08:57:45.063542 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:45.063512    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"]
Nov 03 08:57:45.148894 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:45.148860    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39799a31-d7b7-4948-af12-23cb1f426fc5-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th\" (UID: \"39799a31-d7b7-4948-af12-23cb1f426fc5\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"
Nov 03 08:57:45.149103 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:45.148914    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39799a31-d7b7-4948-af12-23cb1f426fc5-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th\" (UID: \"39799a31-d7b7-4948-af12-23cb1f426fc5\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"
Nov 03 08:57:45.149103 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:45.148973    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39799a31-d7b7-4948-af12-23cb1f426fc5-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th\" (UID: \"39799a31-d7b7-4948-af12-23cb1f426fc5\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"
Nov 03 08:57:45.149103 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:45.149071    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94z2p\" (UniqueName: \"kubernetes.io/projected/39799a31-d7b7-4948-af12-23cb1f426fc5-kube-api-access-94z2p\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th\" (UID: \"39799a31-d7b7-4948-af12-23cb1f426fc5\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"
Nov 03 08:57:45.149282 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:45.149103    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39799a31-d7b7-4948-af12-23cb1f426fc5-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th\" (UID: \"39799a31-d7b7-4948-af12-23cb1f426fc5\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"
Nov 03 08:57:45.149282 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:45.149129    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39799a31-d7b7-4948-af12-23cb1f426fc5-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th\" (UID: \"39799a31-d7b7-4948-af12-23cb1f426fc5\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"
Nov 03 08:57:45.149282 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:45.149164    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39799a31-d7b7-4948-af12-23cb1f426fc5-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th\" (UID: \"39799a31-d7b7-4948-af12-23cb1f426fc5\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"
Nov 03 08:57:45.149282 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:45.149234    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39799a31-d7b7-4948-af12-23cb1f426fc5-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th\" (UID: \"39799a31-d7b7-4948-af12-23cb1f426fc5\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"
Nov 03 08:57:45.149504 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:45.149308    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39799a31-d7b7-4948-af12-23cb1f426fc5-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th\" (UID: \"39799a31-d7b7-4948-af12-23cb1f426fc5\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"
Nov 03 08:57:45.149969 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:45.149909    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39799a31-d7b7-4948-af12-23cb1f426fc5-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th\" (UID: \"39799a31-d7b7-4948-af12-23cb1f426fc5\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"
Nov 03 08:57:45.150081 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:45.149976    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39799a31-d7b7-4948-af12-23cb1f426fc5-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th\" (UID: \"39799a31-d7b7-4948-af12-23cb1f426fc5\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"
Nov 03 08:57:45.150081 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:45.149976    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39799a31-d7b7-4948-af12-23cb1f426fc5-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th\" (UID: \"39799a31-d7b7-4948-af12-23cb1f426fc5\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"
Nov 03 08:57:45.150399 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:45.150371    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39799a31-d7b7-4948-af12-23cb1f426fc5-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th\" (UID: \"39799a31-d7b7-4948-af12-23cb1f426fc5\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"
Nov 03 08:57:45.150498 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:45.150412    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39799a31-d7b7-4948-af12-23cb1f426fc5-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th\" (UID: \"39799a31-d7b7-4948-af12-23cb1f426fc5\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"
Nov 03 08:57:45.152272 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:45.152247    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39799a31-d7b7-4948-af12-23cb1f426fc5-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th\" (UID: \"39799a31-d7b7-4948-af12-23cb1f426fc5\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"
Nov 03 08:57:45.154380 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:45.154352    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39799a31-d7b7-4948-af12-23cb1f426fc5-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th\" (UID: \"39799a31-d7b7-4948-af12-23cb1f426fc5\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"
Nov 03 08:57:45.209159 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:45.209064    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39799a31-d7b7-4948-af12-23cb1f426fc5-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th\" (UID: \"39799a31-d7b7-4948-af12-23cb1f426fc5\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"
Nov 03 08:57:45.209721 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:45.209678    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94z2p\" (UniqueName: \"kubernetes.io/projected/39799a31-d7b7-4948-af12-23cb1f426fc5-kube-api-access-94z2p\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th\" (UID: \"39799a31-d7b7-4948-af12-23cb1f426fc5\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"
Nov 03 08:57:45.278439 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:45.278113    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"
Nov 03 08:57:46.146973 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:46.146136    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"]
Nov 03 08:57:46.307069 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:46.307016    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 08:57:46.307334 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:46.307312    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 08:57:46.307535 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:46.307500    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 08:57:46.734020 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:46.730059    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-whmmw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:57:46.734020 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:46.730123    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw" podUID="14dd6c95-a250-41c5-87c7-ae741f96d01a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:57:47.006284 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:57:47.006135    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:57:47.269223 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:47.265436    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th" event={"ID":"39799a31-d7b7-4948-af12-23cb1f426fc5","Type":"ContainerStarted","Data":"f04fbc63937a5fcdc5a588aebc555a2b4220974320da951445eb70979cf76061"}
Nov 03 08:57:47.269223 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:47.265509    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th" event={"ID":"39799a31-d7b7-4948-af12-23cb1f426fc5","Type":"ContainerStarted","Data":"593c8712c710dd0ba076eba2aef087dfc5c2b1c7f2f561657b9b5d5af2848c57"}
Nov 03 08:57:47.282755 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:47.282554    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"
Nov 03 08:57:47.283873 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:47.283841    4473 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 08:57:47.284148 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:47.284119    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th" podUID="39799a31-d7b7-4948-af12-23cb1f426fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 08:57:47.386094 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:47.386036    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th" podStartSLOduration=3.386015753 podStartE2EDuration="3.386015753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:57:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:57:47.385455761 +0000 UTC m=+1091.654519194" watchObservedRunningTime="2026-11-03 08:57:47.386015753 +0000 UTC m=+1091.655079190"
Nov 03 08:57:48.279019 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:48.278986    4473 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 08:57:48.279411 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:48.279041    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th" podUID="39799a31-d7b7-4948-af12-23cb1f426fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 08:57:49.279574 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:49.279532    4473 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 08:57:49.279990 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:49.279598    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th" podUID="39799a31-d7b7-4948-af12-23cb1f426fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 08:57:50.278971 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:50.278909    4473 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 08:57:50.279154 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:50.279020    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th" podUID="39799a31-d7b7-4948-af12-23cb1f426fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 08:57:51.278732 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:51.278694    4473 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 08:57:51.279144 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:51.278752    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th" podUID="39799a31-d7b7-4948-af12-23cb1f426fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 08:57:52.278886 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:52.278851    4473 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 08:57:52.279321 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:52.278905    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th" podUID="39799a31-d7b7-4948-af12-23cb1f426fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 08:57:53.279436 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:53.279398    4473 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Nov 03 08:57:53.279815 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:53.279454    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th" podUID="39799a31-d7b7-4948-af12-23cb1f426fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Nov 03 08:57:54.283267 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:54.283239    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"
Nov 03 08:57:54.283731 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:54.283624    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"
Nov 03 08:57:54.284386 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:54.284371    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"
Nov 03 08:57:56.456129 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:57:56.456097    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 08:58:16.936256 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:58:16.936165    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:58:17.357905 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:58:17.357831    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 08:58:35.991061 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:58:35.991034    4473 scope.go:122] "RemoveContainer" containerID="a8f9d1a0f43f94b53da8892599c834432580a548bdf7ce354336ed14886b632d"
Nov 03 08:58:42.610810 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:58:42.610774    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 08:58:46.937904 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:58:46.937869    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:58:48.753136 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:58:48.753108    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-910d50/e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf"]
Nov 03 08:58:48.758056 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:58:48.758036    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf"
Nov 03 08:58:48.862406 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:58:48.862380    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-910d50\"/\"e2e-conflict-b-54a1bbf9-kserve-self-signed-certs\""
Nov 03 08:58:48.864858 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:58:48.864837    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-910d50/e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf"]
Nov 03 08:58:48.873549 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:58:48.873527    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-tmp-dir\") pod \"e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf\" (UID: \"d26a0519-b53f-4b0a-881d-1bcb2e10b8d5\") " pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf"
Nov 03 08:58:48.873610 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:58:48.873580    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-model-cache\") pod \"e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf\" (UID: \"d26a0519-b53f-4b0a-881d-1bcb2e10b8d5\") " pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf"
Nov 03 08:58:48.873682 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:58:48.873666    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-tls-certs\") pod \"e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf\" (UID: \"d26a0519-b53f-4b0a-881d-1bcb2e10b8d5\") " pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf"
Nov 03 08:58:48.873744 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:58:48.873730    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5w5l\" (UniqueName: \"kubernetes.io/projected/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-kube-api-access-h5w5l\") pod \"e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf\" (UID: \"d26a0519-b53f-4b0a-881d-1bcb2e10b8d5\") " pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf"
Nov 03 08:58:48.873785 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:58:48.873765    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-home\") pod \"e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf\" (UID: \"d26a0519-b53f-4b0a-881d-1bcb2e10b8d5\") " pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf"
Nov 03 08:58:48.873827 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:58:48.873808    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-dshm\") pod \"e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf\" (UID: \"d26a0519-b53f-4b0a-881d-1bcb2e10b8d5\") " pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf"
Nov 03 08:58:48.975570 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:58:48.975531    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-model-cache\") pod \"e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf\" (UID: \"d26a0519-b53f-4b0a-881d-1bcb2e10b8d5\") " pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf"
Nov 03 08:58:48.975744 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:58:48.975611    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-tls-certs\") pod \"e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf\" (UID: \"d26a0519-b53f-4b0a-881d-1bcb2e10b8d5\") " pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf"
Nov 03 08:58:48.975744 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:58:48.975705    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5w5l\" (UniqueName: \"kubernetes.io/projected/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-kube-api-access-h5w5l\") pod \"e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf\" (UID: \"d26a0519-b53f-4b0a-881d-1bcb2e10b8d5\") " pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf"
Nov 03 08:58:48.975744 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:58:48.975736    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-home\") pod \"e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf\" (UID: \"d26a0519-b53f-4b0a-881d-1bcb2e10b8d5\") " pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf"
Nov 03 08:58:48.975940 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:58:48.975758    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-dshm\") pod \"e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf\" (UID: \"d26a0519-b53f-4b0a-881d-1bcb2e10b8d5\") " pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf"
Nov 03 08:58:48.975940 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:58:48.975853    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-tmp-dir\") pod \"e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf\" (UID: \"d26a0519-b53f-4b0a-881d-1bcb2e10b8d5\") " pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf"
Nov 03 08:58:48.976071 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:58:48.976009    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-model-cache\") pod \"e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf\" (UID: \"d26a0519-b53f-4b0a-881d-1bcb2e10b8d5\") " pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf"
Nov 03 08:58:48.976134 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:58:48.976110    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-home\") pod \"e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf\" (UID: \"d26a0519-b53f-4b0a-881d-1bcb2e10b8d5\") " pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf"
Nov 03 08:58:48.976224 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:58:48.976201    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-tmp-dir\") pod \"e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf\" (UID: \"d26a0519-b53f-4b0a-881d-1bcb2e10b8d5\") " pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf"
Nov 03 08:58:48.980995 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:58:48.980975    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-dshm\") pod \"e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf\" (UID: \"d26a0519-b53f-4b0a-881d-1bcb2e10b8d5\") " pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf"
Nov 03 08:58:48.981267 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:58:48.981249    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-tls-certs\") pod \"e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf\" (UID: \"d26a0519-b53f-4b0a-881d-1bcb2e10b8d5\") " pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf"
Nov 03 08:58:49.037758 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:58:49.037694    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5w5l\" (UniqueName: \"kubernetes.io/projected/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-kube-api-access-h5w5l\") pod \"e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf\" (UID: \"d26a0519-b53f-4b0a-881d-1bcb2e10b8d5\") " pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf"
Nov 03 08:58:49.069791 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:58:49.069764    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf"
Nov 03 08:58:49.467714 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:58:49.467685    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-910d50/e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf"]
Nov 03 08:58:49.469093 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:58:49.469065    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd26a0519_b53f_4b0a_881d_1bcb2e10b8d5.slice/crio-0ff11887de177980b9411256a45718abfd2f67c892caba5a09b4c53754cf580e WatchSource:0}: Error finding container 0ff11887de177980b9411256a45718abfd2f67c892caba5a09b4c53754cf580e: Status 404 returned error can't find the container with id 0ff11887de177980b9411256a45718abfd2f67c892caba5a09b4c53754cf580e
Nov 03 08:58:49.725487 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:58:49.725411    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf" event={"ID":"d26a0519-b53f-4b0a-881d-1bcb2e10b8d5","Type":"ContainerStarted","Data":"5f5560fd4725dab1b9175d43ae43590103745ab1bc14673883828f02a66fde76"}
Nov 03 08:58:49.725487 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:58:49.725455    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf" event={"ID":"d26a0519-b53f-4b0a-881d-1bcb2e10b8d5","Type":"ContainerStarted","Data":"0ff11887de177980b9411256a45718abfd2f67c892caba5a09b4c53754cf580e"}
Nov 03 08:58:49.843967 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:58:49.843904    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf" podStartSLOduration=1.843888711 podStartE2EDuration="1.843888711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:58:49.84221621 +0000 UTC m=+1154.111279670" watchObservedRunningTime="2026-11-03 08:58:49.843888711 +0000 UTC m=+1154.112952182"
Nov 03 08:58:50.070034 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:58:50.069886    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf"
Nov 03 08:58:59.070560 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:58:59.070528    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf"
Nov 03 08:58:59.082914 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:58:59.082892    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf"
Nov 03 08:58:59.792997 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:58:59.792938    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf"
Nov 03 08:59:12.764312 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:12.764277    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 08:59:16.937556 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:59:16.937519    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:59:34.716244 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:34.716210    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 08:59:36.015250 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:36.015216    4473 scope.go:122] "RemoveContainer" containerID="26a41d8cd2ec15af243332318d488ab2a19451be7fda333d74178eeb8ac12a60"
Nov 03 08:59:36.019385 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:36.019364    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 08:59:36.021244 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:36.021226    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 08:59:36.025909 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:36.025887    4473 scope.go:122] "RemoveContainer" containerID="65ebd9aea50140772826a5642b72418603ee63eb9b85c77f72616c2eb8fd6e99"
Nov 03 08:59:36.035521 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:36.035503    4473 scope.go:122] "RemoveContainer" containerID="dab4271f861945cd0862a2b9825165397fa64db79baf50f14978d7f23cf8343c"
Nov 03 08:59:42.395355 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:42.395323    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn"]
Nov 03 08:59:42.400418 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:42.400399    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn"
Nov 03 08:59:42.463128 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:42.463101    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05a8a676-3011-4597-9b71-b70c3387e2b9-home\") pod \"distinct-2-w1-26e861-kserve-54ddb996c-2rxxn\" (UID: \"05a8a676-3011-4597-9b71-b70c3387e2b9\") " pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn"
Nov 03 08:59:42.463299 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:42.463134    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05a8a676-3011-4597-9b71-b70c3387e2b9-tls-certs\") pod \"distinct-2-w1-26e861-kserve-54ddb996c-2rxxn\" (UID: \"05a8a676-3011-4597-9b71-b70c3387e2b9\") " pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn"
Nov 03 08:59:42.463299 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:42.463153    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05a8a676-3011-4597-9b71-b70c3387e2b9-model-cache\") pod \"distinct-2-w1-26e861-kserve-54ddb996c-2rxxn\" (UID: \"05a8a676-3011-4597-9b71-b70c3387e2b9\") " pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn"
Nov 03 08:59:42.463299 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:42.463248    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05a8a676-3011-4597-9b71-b70c3387e2b9-tmp-dir\") pod \"distinct-2-w1-26e861-kserve-54ddb996c-2rxxn\" (UID: \"05a8a676-3011-4597-9b71-b70c3387e2b9\") " pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn"
Nov 03 08:59:42.463423 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:42.463380    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbxzp\" (UniqueName: \"kubernetes.io/projected/05a8a676-3011-4597-9b71-b70c3387e2b9-kube-api-access-nbxzp\") pod \"distinct-2-w1-26e861-kserve-54ddb996c-2rxxn\" (UID: \"05a8a676-3011-4597-9b71-b70c3387e2b9\") " pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn"
Nov 03 08:59:42.463482 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:42.463442    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05a8a676-3011-4597-9b71-b70c3387e2b9-dshm\") pod \"distinct-2-w1-26e861-kserve-54ddb996c-2rxxn\" (UID: \"05a8a676-3011-4597-9b71-b70c3387e2b9\") " pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn"
Nov 03 08:59:42.469989 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:42.469961    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn"]
Nov 03 08:59:42.505374 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:42.505346    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-26e861\"/\"openshift-service-ca.crt\""
Nov 03 08:59:42.505374 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:42.505349    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-26e861\"/\"kube-root-ca.crt\""
Nov 03 08:59:42.505533 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:42.505371    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-26e861\"/\"default-dockercfg-hl2h5\""
Nov 03 08:59:42.505608 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:42.505591    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-26e861\"/\"distinct-2-w1-26e861-kserve-self-signed-certs\""
Nov 03 08:59:42.567974 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:42.566659    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05a8a676-3011-4597-9b71-b70c3387e2b9-dshm\") pod \"distinct-2-w1-26e861-kserve-54ddb996c-2rxxn\" (UID: \"05a8a676-3011-4597-9b71-b70c3387e2b9\") " pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn"
Nov 03 08:59:42.567974 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:42.567319    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05a8a676-3011-4597-9b71-b70c3387e2b9-home\") pod \"distinct-2-w1-26e861-kserve-54ddb996c-2rxxn\" (UID: \"05a8a676-3011-4597-9b71-b70c3387e2b9\") " pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn"
Nov 03 08:59:42.567974 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:42.567666    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05a8a676-3011-4597-9b71-b70c3387e2b9-home\") pod \"distinct-2-w1-26e861-kserve-54ddb996c-2rxxn\" (UID: \"05a8a676-3011-4597-9b71-b70c3387e2b9\") " pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn"
Nov 03 08:59:42.567974 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:42.567732    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05a8a676-3011-4597-9b71-b70c3387e2b9-tls-certs\") pod \"distinct-2-w1-26e861-kserve-54ddb996c-2rxxn\" (UID: \"05a8a676-3011-4597-9b71-b70c3387e2b9\") " pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn"
Nov 03 08:59:42.567974 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:42.567764    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05a8a676-3011-4597-9b71-b70c3387e2b9-model-cache\") pod \"distinct-2-w1-26e861-kserve-54ddb996c-2rxxn\" (UID: \"05a8a676-3011-4597-9b71-b70c3387e2b9\") " pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn"
Nov 03 08:59:42.567974 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:42.567815    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05a8a676-3011-4597-9b71-b70c3387e2b9-tmp-dir\") pod \"distinct-2-w1-26e861-kserve-54ddb996c-2rxxn\" (UID: \"05a8a676-3011-4597-9b71-b70c3387e2b9\") " pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn"
Nov 03 08:59:42.567974 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:42.567955    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbxzp\" (UniqueName: \"kubernetes.io/projected/05a8a676-3011-4597-9b71-b70c3387e2b9-kube-api-access-nbxzp\") pod \"distinct-2-w1-26e861-kserve-54ddb996c-2rxxn\" (UID: \"05a8a676-3011-4597-9b71-b70c3387e2b9\") " pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn"
Nov 03 08:59:42.569135 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:42.569054    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05a8a676-3011-4597-9b71-b70c3387e2b9-tmp-dir\") pod \"distinct-2-w1-26e861-kserve-54ddb996c-2rxxn\" (UID: \"05a8a676-3011-4597-9b71-b70c3387e2b9\") " pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn"
Nov 03 08:59:42.571173 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:42.569554    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05a8a676-3011-4597-9b71-b70c3387e2b9-model-cache\") pod \"distinct-2-w1-26e861-kserve-54ddb996c-2rxxn\" (UID: \"05a8a676-3011-4597-9b71-b70c3387e2b9\") " pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn"
Nov 03 08:59:42.571539 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:42.571520    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05a8a676-3011-4597-9b71-b70c3387e2b9-dshm\") pod \"distinct-2-w1-26e861-kserve-54ddb996c-2rxxn\" (UID: \"05a8a676-3011-4597-9b71-b70c3387e2b9\") " pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn"
Nov 03 08:59:42.578621 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:42.578579    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05a8a676-3011-4597-9b71-b70c3387e2b9-tls-certs\") pod \"distinct-2-w1-26e861-kserve-54ddb996c-2rxxn\" (UID: \"05a8a676-3011-4597-9b71-b70c3387e2b9\") " pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn"
Nov 03 08:59:42.627782 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:42.627749    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbxzp\" (UniqueName: \"kubernetes.io/projected/05a8a676-3011-4597-9b71-b70c3387e2b9-kube-api-access-nbxzp\") pod \"distinct-2-w1-26e861-kserve-54ddb996c-2rxxn\" (UID: \"05a8a676-3011-4597-9b71-b70c3387e2b9\") " pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn"
Nov 03 08:59:42.712767 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:42.712678    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn"
Nov 03 08:59:43.176785 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:43.176762    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn"]
Nov 03 08:59:43.177992 ip-10-0-139-73 kubenswrapper[4473]: W1103 08:59:43.177969    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05a8a676_3011_4597_9b71_b70c3387e2b9.slice/crio-4656d7502ceea8c6c8903036d9a0a7cb164574383eb5cb1e3d1969953ee0c95b WatchSource:0}: Error finding container 4656d7502ceea8c6c8903036d9a0a7cb164574383eb5cb1e3d1969953ee0c95b: Status 404 returned error can't find the container with id 4656d7502ceea8c6c8903036d9a0a7cb164574383eb5cb1e3d1969953ee0c95b
Nov 03 08:59:44.032930 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:44.032892    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn" event={"ID":"05a8a676-3011-4597-9b71-b70c3387e2b9","Type":"ContainerStarted","Data":"6490c4d695eff37244ab1ae606833c5f7b7630aa47932063b70f96f73129ea42"}
Nov 03 08:59:44.033352 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:44.032970    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn" event={"ID":"05a8a676-3011-4597-9b71-b70c3387e2b9","Type":"ContainerStarted","Data":"4656d7502ceea8c6c8903036d9a0a7cb164574383eb5cb1e3d1969953ee0c95b"}
Nov 03 08:59:44.151379 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:44.151337    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn" podStartSLOduration=2.151322818 podStartE2EDuration="2.151322818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:59:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:59:44.149158563 +0000 UTC m=+1208.418222023" watchObservedRunningTime="2026-11-03 08:59:44.151322818 +0000 UTC m=+1208.420386251"
Nov 03 08:59:44.713551 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:44.713516    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn"
Nov 03 08:59:46.938022 ip-10-0-139-73 kubenswrapper[4473]: E1103 08:59:46.937917    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 08:59:52.713771 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:52.713740    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn"
Nov 03 08:59:52.726340 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:52.726317    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn"
Nov 03 08:59:53.093119 ip-10-0-139-73 kubenswrapper[4473]: I1103 08:59:53.093051    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn"
Nov 03 09:00:00.180785 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:00.180747    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894940-gzdxr"]
Nov 03 09:00:00.184638 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:00.184619    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894940-phzwc"]
Nov 03 09:00:00.184788 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:00.184764    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894940-gzdxr"
Nov 03 09:00:00.188464 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:00.188445    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894940-2wdhm"]
Nov 03 09:00:00.188573 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:00.188547    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894940-phzwc"
Nov 03 09:00:00.192340 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:00.192324    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894940-2wdhm"
Nov 03 09:00:00.204261 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:00.204239    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:00:00.239935 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:00.239902    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:00:00.239935 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:00.239906    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:00:00.240172 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:00.239999    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:00:00.280088 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:00.280056    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdpdp\" (UniqueName: \"kubernetes.io/projected/b090afa5-9f65-4d79-b11b-abd7f02b03cc-kube-api-access-zdpdp\") pod \"maas-api-key-cleanup-e2e-worker-w4-18f9da-29894940-phzwc\" (UID: \"b090afa5-9f65-4d79-b11b-abd7f02b03cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894940-phzwc"
Nov 03 09:00:00.280088 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:00.280093    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8sbb\" (UniqueName: \"kubernetes.io/projected/2967ad42-d177-44c0-bdbb-27818c7a44cf-kube-api-access-k8sbb\") pod \"maas-api-key-cleanup-e2e-worker-w0-910d50-29894940-2wdhm\" (UID: \"2967ad42-d177-44c0-bdbb-27818c7a44cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894940-2wdhm"
Nov 03 09:00:00.280312 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:00.280119    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2967ad42-d177-44c0-bdbb-27818c7a44cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-910d50-29894940-2wdhm\" (UID: \"2967ad42-d177-44c0-bdbb-27818c7a44cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894940-2wdhm"
Nov 03 09:00:00.280312 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:00.280260    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7fc3bdf-392e-4042-a0af-aed703adc65c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-77207e-29894940-gzdxr\" (UID: \"b7fc3bdf-392e-4042-a0af-aed703adc65c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894940-gzdxr"
Nov 03 09:00:00.280312 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:00.280290    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b090afa5-9f65-4d79-b11b-abd7f02b03cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-18f9da-29894940-phzwc\" (UID: \"b090afa5-9f65-4d79-b11b-abd7f02b03cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894940-phzwc"
Nov 03 09:00:00.280430 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:00.280370    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfglf\" (UniqueName: \"kubernetes.io/projected/b7fc3bdf-392e-4042-a0af-aed703adc65c-kube-api-access-bfglf\") pod \"maas-api-key-cleanup-e2e-worker-w2-77207e-29894940-gzdxr\" (UID: \"b7fc3bdf-392e-4042-a0af-aed703adc65c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894940-gzdxr"
Nov 03 09:00:00.284910 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:00.284737    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894940-gzdxr"]
Nov 03 09:00:00.287910 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:00.287888    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894940-phzwc"]
Nov 03 09:00:00.354390 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:00.354358    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894940-2wdhm"]
Nov 03 09:00:00.381032 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:00.380998    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdpdp\" (UniqueName: \"kubernetes.io/projected/b090afa5-9f65-4d79-b11b-abd7f02b03cc-kube-api-access-zdpdp\") pod \"maas-api-key-cleanup-e2e-worker-w4-18f9da-29894940-phzwc\" (UID: \"b090afa5-9f65-4d79-b11b-abd7f02b03cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894940-phzwc"
Nov 03 09:00:00.381032 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:00.381036    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8sbb\" (UniqueName: \"kubernetes.io/projected/2967ad42-d177-44c0-bdbb-27818c7a44cf-kube-api-access-k8sbb\") pod \"maas-api-key-cleanup-e2e-worker-w0-910d50-29894940-2wdhm\" (UID: \"2967ad42-d177-44c0-bdbb-27818c7a44cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894940-2wdhm"
Nov 03 09:00:00.381287 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:00.381068    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2967ad42-d177-44c0-bdbb-27818c7a44cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-910d50-29894940-2wdhm\" (UID: \"2967ad42-d177-44c0-bdbb-27818c7a44cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894940-2wdhm"
Nov 03 09:00:00.381287 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:00.381116    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7fc3bdf-392e-4042-a0af-aed703adc65c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-77207e-29894940-gzdxr\" (UID: \"b7fc3bdf-392e-4042-a0af-aed703adc65c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894940-gzdxr"
Nov 03 09:00:00.381406 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:00.381311    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b090afa5-9f65-4d79-b11b-abd7f02b03cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-18f9da-29894940-phzwc\" (UID: \"b090afa5-9f65-4d79-b11b-abd7f02b03cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894940-phzwc"
Nov 03 09:00:00.381465 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:00.381396    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfglf\" (UniqueName: \"kubernetes.io/projected/b7fc3bdf-392e-4042-a0af-aed703adc65c-kube-api-access-bfglf\") pod \"maas-api-key-cleanup-e2e-worker-w2-77207e-29894940-gzdxr\" (UID: \"b7fc3bdf-392e-4042-a0af-aed703adc65c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894940-gzdxr"
Nov 03 09:00:00.381801 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:00.381778    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2967ad42-d177-44c0-bdbb-27818c7a44cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-910d50-29894940-2wdhm\" (UID: \"2967ad42-d177-44c0-bdbb-27818c7a44cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894940-2wdhm"
Nov 03 09:00:00.382195 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:00.382170    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7fc3bdf-392e-4042-a0af-aed703adc65c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-77207e-29894940-gzdxr\" (UID: \"b7fc3bdf-392e-4042-a0af-aed703adc65c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894940-gzdxr"
Nov 03 09:00:00.382936 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:00.382915    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b090afa5-9f65-4d79-b11b-abd7f02b03cc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-18f9da-29894940-phzwc\" (UID: \"b090afa5-9f65-4d79-b11b-abd7f02b03cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894940-phzwc"
Nov 03 09:00:00.448703 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:00.448629    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8sbb\" (UniqueName: \"kubernetes.io/projected/2967ad42-d177-44c0-bdbb-27818c7a44cf-kube-api-access-k8sbb\") pod \"maas-api-key-cleanup-e2e-worker-w0-910d50-29894940-2wdhm\" (UID: \"2967ad42-d177-44c0-bdbb-27818c7a44cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894940-2wdhm"
Nov 03 09:00:00.451828 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:00.451796    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfglf\" (UniqueName: \"kubernetes.io/projected/b7fc3bdf-392e-4042-a0af-aed703adc65c-kube-api-access-bfglf\") pod \"maas-api-key-cleanup-e2e-worker-w2-77207e-29894940-gzdxr\" (UID: \"b7fc3bdf-392e-4042-a0af-aed703adc65c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894940-gzdxr"
Nov 03 09:00:00.452570 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:00.452553    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdpdp\" (UniqueName: \"kubernetes.io/projected/b090afa5-9f65-4d79-b11b-abd7f02b03cc-kube-api-access-zdpdp\") pod \"maas-api-key-cleanup-e2e-worker-w4-18f9da-29894940-phzwc\" (UID: \"b090afa5-9f65-4d79-b11b-abd7f02b03cc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894940-phzwc"
Nov 03 09:00:00.506127 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:00.505827    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894940-gzdxr"
Nov 03 09:00:00.510542 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:00.510507    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894940-phzwc"
Nov 03 09:00:00.519403 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:00.519375    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894940-2wdhm"
Nov 03 09:00:01.028830 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:01.028804    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894940-gzdxr"]
Nov 03 09:00:01.029896 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:00:01.029868    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7fc3bdf_392e_4042_a0af_aed703adc65c.slice/crio-c29c6b59e98db482047f647087865fd826a2b35a86cef2128a46601f95efd8de WatchSource:0}: Error finding container c29c6b59e98db482047f647087865fd826a2b35a86cef2128a46601f95efd8de: Status 404 returned error can't find the container with id c29c6b59e98db482047f647087865fd826a2b35a86cef2128a46601f95efd8de
Nov 03 09:00:01.084443 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:01.084414    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894940-phzwc"]
Nov 03 09:00:01.085752 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:00:01.085722    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb090afa5_9f65_4d79_b11b_abd7f02b03cc.slice/crio-bea4b6c861cf67d2f6a4394fd5894bc1b86167d90d54cdc7bafc4afa37336e48 WatchSource:0}: Error finding container bea4b6c861cf67d2f6a4394fd5894bc1b86167d90d54cdc7bafc4afa37336e48: Status 404 returned error can't find the container with id bea4b6c861cf67d2f6a4394fd5894bc1b86167d90d54cdc7bafc4afa37336e48
Nov 03 09:00:01.125755 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:01.125726    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894940-gzdxr" event={"ID":"b7fc3bdf-392e-4042-a0af-aed703adc65c","Type":"ContainerStarted","Data":"c29c6b59e98db482047f647087865fd826a2b35a86cef2128a46601f95efd8de"}
Nov 03 09:00:01.127008 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:01.126982    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894940-phzwc" event={"ID":"b090afa5-9f65-4d79-b11b-abd7f02b03cc","Type":"ContainerStarted","Data":"bea4b6c861cf67d2f6a4394fd5894bc1b86167d90d54cdc7bafc4afa37336e48"}
Nov 03 09:00:01.133633 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:01.133613    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894940-2wdhm"]
Nov 03 09:00:01.134961 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:00:01.134922    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2967ad42_d177_44c0_bdbb_27818c7a44cf.slice/crio-5c9f1632c232be18d27a7e5c3ebfedc20bfa6fcab4cb70415fe5eb6b60e5bf72 WatchSource:0}: Error finding container 5c9f1632c232be18d27a7e5c3ebfedc20bfa6fcab4cb70415fe5eb6b60e5bf72: Status 404 returned error can't find the container with id 5c9f1632c232be18d27a7e5c3ebfedc20bfa6fcab4cb70415fe5eb6b60e5bf72
Nov 03 09:00:02.136281 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:02.136231    4473 generic.go:356] "Generic (PLEG): container finished" podID="b7fc3bdf-392e-4042-a0af-aed703adc65c" containerID="da1381831a96fa265134ea45a7260ed453bf0a357ab85b019f338fff4a815a0f" exitCode=60
Nov 03 09:00:02.136670 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:02.136341    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894940-gzdxr" event={"ID":"b7fc3bdf-392e-4042-a0af-aed703adc65c","Type":"ContainerDied","Data":"da1381831a96fa265134ea45a7260ed453bf0a357ab85b019f338fff4a815a0f"}
Nov 03 09:00:02.136791 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:02.136770    4473 scope.go:122] "RemoveContainer" containerID="da1381831a96fa265134ea45a7260ed453bf0a357ab85b019f338fff4a815a0f"
Nov 03 09:00:02.139830 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:02.139809    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894940-phzwc" event={"ID":"b090afa5-9f65-4d79-b11b-abd7f02b03cc","Type":"ContainerStarted","Data":"90c15466a7801697444566d6d3cb96779ae514ae39274094a323ced9e554c458"}
Nov 03 09:00:02.140082 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:02.140066    4473 scope.go:122] "RemoveContainer" containerID="90c15466a7801697444566d6d3cb96779ae514ae39274094a323ced9e554c458"
Nov 03 09:00:02.141684 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:02.141660    4473 generic.go:356] "Generic (PLEG): container finished" podID="2967ad42-d177-44c0-bdbb-27818c7a44cf" containerID="5c5e4afa093fd6b3c166c325ee6a55d167595f6fe6e267866b3496a36f3dc26e" exitCode=60
Nov 03 09:00:02.141789 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:02.141698    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894940-2wdhm" event={"ID":"2967ad42-d177-44c0-bdbb-27818c7a44cf","Type":"ContainerDied","Data":"5c5e4afa093fd6b3c166c325ee6a55d167595f6fe6e267866b3496a36f3dc26e"}
Nov 03 09:00:02.141789 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:02.141738    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894940-2wdhm" event={"ID":"2967ad42-d177-44c0-bdbb-27818c7a44cf","Type":"ContainerStarted","Data":"5c9f1632c232be18d27a7e5c3ebfedc20bfa6fcab4cb70415fe5eb6b60e5bf72"}
Nov 03 09:00:02.141918 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:02.141905    4473 scope.go:122] "RemoveContainer" containerID="5c5e4afa093fd6b3c166c325ee6a55d167595f6fe6e267866b3496a36f3dc26e"
Nov 03 09:00:03.149124 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:03.149095    4473 generic.go:356] "Generic (PLEG): container finished" podID="2967ad42-d177-44c0-bdbb-27818c7a44cf" containerID="2b757546955b05bb4f26a62b8caf0bcd4c778b5f6a84cdec5bb2644eb38af3d4" exitCode=60
Nov 03 09:00:03.149506 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:03.149173    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894940-2wdhm" event={"ID":"2967ad42-d177-44c0-bdbb-27818c7a44cf","Type":"ContainerDied","Data":"2b757546955b05bb4f26a62b8caf0bcd4c778b5f6a84cdec5bb2644eb38af3d4"}
Nov 03 09:00:03.149506 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:03.149228    4473 scope.go:122] "RemoveContainer" containerID="5c5e4afa093fd6b3c166c325ee6a55d167595f6fe6e267866b3496a36f3dc26e"
Nov 03 09:00:03.149506 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:03.149461    4473 scope.go:122] "RemoveContainer" containerID="2b757546955b05bb4f26a62b8caf0bcd4c778b5f6a84cdec5bb2644eb38af3d4"
Nov 03 09:00:03.149796 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:00:03.149773    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-910d50-29894940-2wdhm_odh-ai-gateway-infra(2967ad42-d177-44c0-bdbb-27818c7a44cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894940-2wdhm" podUID="2967ad42-d177-44c0-bdbb-27818c7a44cf"
Nov 03 09:00:03.151241 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:03.151220    4473 generic.go:356] "Generic (PLEG): container finished" podID="b7fc3bdf-392e-4042-a0af-aed703adc65c" containerID="9e2fa84fdc0f01b94c274d8bc4e11211aee8c9c06587d08131e1f6e5cdc208f0" exitCode=60
Nov 03 09:00:03.151359 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:03.151300    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894940-gzdxr" event={"ID":"b7fc3bdf-392e-4042-a0af-aed703adc65c","Type":"ContainerDied","Data":"9e2fa84fdc0f01b94c274d8bc4e11211aee8c9c06587d08131e1f6e5cdc208f0"}
Nov 03 09:00:03.151550 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:03.151536    4473 scope.go:122] "RemoveContainer" containerID="9e2fa84fdc0f01b94c274d8bc4e11211aee8c9c06587d08131e1f6e5cdc208f0"
Nov 03 09:00:03.151810 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:00:03.151786    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-77207e-29894940-gzdxr_odh-ai-gateway-infra(b7fc3bdf-392e-4042-a0af-aed703adc65c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894940-gzdxr" podUID="b7fc3bdf-392e-4042-a0af-aed703adc65c"
Nov 03 09:00:03.153256 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:03.153214    4473 generic.go:356] "Generic (PLEG): container finished" podID="b090afa5-9f65-4d79-b11b-abd7f02b03cc" containerID="90c15466a7801697444566d6d3cb96779ae514ae39274094a323ced9e554c458" exitCode=60
Nov 03 09:00:03.153256 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:03.153236    4473 generic.go:356] "Generic (PLEG): container finished" podID="b090afa5-9f65-4d79-b11b-abd7f02b03cc" containerID="a1a52214f11364e349169c29e8b4f15aeabad1a4fca8c444f81e5b16ca5c8f78" exitCode=60
Nov 03 09:00:03.153493 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:03.153313    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894940-phzwc" event={"ID":"b090afa5-9f65-4d79-b11b-abd7f02b03cc","Type":"ContainerDied","Data":"90c15466a7801697444566d6d3cb96779ae514ae39274094a323ced9e554c458"}
Nov 03 09:00:03.153493 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:03.153347    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894940-phzwc" event={"ID":"b090afa5-9f65-4d79-b11b-abd7f02b03cc","Type":"ContainerDied","Data":"a1a52214f11364e349169c29e8b4f15aeabad1a4fca8c444f81e5b16ca5c8f78"}
Nov 03 09:00:03.153723 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:03.153703    4473 scope.go:122] "RemoveContainer" containerID="a1a52214f11364e349169c29e8b4f15aeabad1a4fca8c444f81e5b16ca5c8f78"
Nov 03 09:00:03.154003 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:00:03.153982    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-18f9da-29894940-phzwc_odh-ai-gateway-infra(b090afa5-9f65-4d79-b11b-abd7f02b03cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894940-phzwc" podUID="b090afa5-9f65-4d79-b11b-abd7f02b03cc"
Nov 03 09:00:03.172279 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:03.172256    4473 scope.go:122] "RemoveContainer" containerID="da1381831a96fa265134ea45a7260ed453bf0a357ab85b019f338fff4a815a0f"
Nov 03 09:00:03.186525 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:03.186504    4473 scope.go:122] "RemoveContainer" containerID="90c15466a7801697444566d6d3cb96779ae514ae39274094a323ced9e554c458"
Nov 03 09:00:03.198749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:03.198727    4473 scope.go:122] "RemoveContainer" containerID="90c15466a7801697444566d6d3cb96779ae514ae39274094a323ced9e554c458"
Nov 03 09:00:03.199082 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:00:03.199058    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90c15466a7801697444566d6d3cb96779ae514ae39274094a323ced9e554c458\": container with ID starting with 90c15466a7801697444566d6d3cb96779ae514ae39274094a323ced9e554c458 not found: ID does not exist" containerID="90c15466a7801697444566d6d3cb96779ae514ae39274094a323ced9e554c458"
Nov 03 09:00:03.199171 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:03.199090    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90c15466a7801697444566d6d3cb96779ae514ae39274094a323ced9e554c458"} err="failed to get container status \"90c15466a7801697444566d6d3cb96779ae514ae39274094a323ced9e554c458\": rpc error: code = NotFound desc = could not find container \"90c15466a7801697444566d6d3cb96779ae514ae39274094a323ced9e554c458\": container with ID starting with 90c15466a7801697444566d6d3cb96779ae514ae39274094a323ced9e554c458 not found: ID does not exist"
Nov 03 09:00:04.161503 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:04.161474    4473 scope.go:122] "RemoveContainer" containerID="2b757546955b05bb4f26a62b8caf0bcd4c778b5f6a84cdec5bb2644eb38af3d4"
Nov 03 09:00:04.161986 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:00:04.161758    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-910d50-29894940-2wdhm_odh-ai-gateway-infra(2967ad42-d177-44c0-bdbb-27818c7a44cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894940-2wdhm" podUID="2967ad42-d177-44c0-bdbb-27818c7a44cf"
Nov 03 09:00:04.163097 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:04.163075    4473 scope.go:122] "RemoveContainer" containerID="9e2fa84fdc0f01b94c274d8bc4e11211aee8c9c06587d08131e1f6e5cdc208f0"
Nov 03 09:00:04.163356 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:00:04.163329    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-77207e-29894940-gzdxr_odh-ai-gateway-infra(b7fc3bdf-392e-4042-a0af-aed703adc65c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894940-gzdxr" podUID="b7fc3bdf-392e-4042-a0af-aed703adc65c"
Nov 03 09:00:04.164602 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:04.164586    4473 scope.go:122] "RemoveContainer" containerID="a1a52214f11364e349169c29e8b4f15aeabad1a4fca8c444f81e5b16ca5c8f78"
Nov 03 09:00:04.164790 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:00:04.164772    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-18f9da-29894940-phzwc_odh-ai-gateway-infra(b090afa5-9f65-4d79-b11b-abd7f02b03cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894940-phzwc" podUID="b090afa5-9f65-4d79-b11b-abd7f02b03cc"
Nov 03 09:00:08.063484 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:08.063452    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d76c0807-gt9d5"]
Nov 03 09:00:08.113599 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:08.113568    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d76c0807-gt9d5"]
Nov 03 09:00:09.869399 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:09.869368    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="490bbf6e-ce79-48de-8357-eb0f1c19055e" path="/var/lib/kubelet/pods/490bbf6e-ce79-48de-8357-eb0f1c19055e/volumes"
Nov 03 09:00:13.863098 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:13.863063    4473 scope.go:122] "RemoveContainer" containerID="a1a52214f11364e349169c29e8b4f15aeabad1a4fca8c444f81e5b16ca5c8f78"
Nov 03 09:00:13.863500 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:13.863240    4473 scope.go:122] "RemoveContainer" containerID="2b757546955b05bb4f26a62b8caf0bcd4c778b5f6a84cdec5bb2644eb38af3d4"
Nov 03 09:00:14.223684 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:14.223653    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894940-phzwc" event={"ID":"b090afa5-9f65-4d79-b11b-abd7f02b03cc","Type":"ContainerStarted","Data":"ceb6a7d5dd4b68c77c1d75049cb33adf9b2b3b2c991b223a628334d3f763dd97"}
Nov 03 09:00:14.225687 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:14.225664    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894940-2wdhm" event={"ID":"2967ad42-d177-44c0-bdbb-27818c7a44cf","Type":"ContainerStarted","Data":"4cef6b493ad5f91903650ff52b0d817ccf42ee5dfe97b3a499a304645ab7efa3"}
Nov 03 09:00:14.347255 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:14.347210    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894940-phzwc" podStartSLOduration=14.347196538 podStartE2EDuration="14.347196538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:00:14.345219936 +0000 UTC m=+1238.614283398" watchObservedRunningTime="2026-11-03 09:00:14.347196538 +0000 UTC m=+1238.616259972"
Nov 03 09:00:14.463586 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:14.463490    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894940-2wdhm" podStartSLOduration=14.46347633 podStartE2EDuration="14.46347633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:00:14.460652487 +0000 UTC m=+1238.729715944" watchObservedRunningTime="2026-11-03 09:00:14.46347633 +0000 UTC m=+1238.732539763"
Nov 03 09:00:14.863077 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:14.862991    4473 scope.go:122] "RemoveContainer" containerID="9e2fa84fdc0f01b94c274d8bc4e11211aee8c9c06587d08131e1f6e5cdc208f0"
Nov 03 09:00:15.233487 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:15.233446    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894940-gzdxr" event={"ID":"b7fc3bdf-392e-4042-a0af-aed703adc65c","Type":"ContainerStarted","Data":"c1583df594436e6ca094b070e66052dda8a406fcdc19e41d7ea2e9309bf2e1b6"}
Nov 03 09:00:15.235344 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:15.235321    4473 generic.go:356] "Generic (PLEG): container finished" podID="b090afa5-9f65-4d79-b11b-abd7f02b03cc" containerID="ceb6a7d5dd4b68c77c1d75049cb33adf9b2b3b2c991b223a628334d3f763dd97" exitCode=60
Nov 03 09:00:15.235468 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:15.235397    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894940-phzwc" event={"ID":"b090afa5-9f65-4d79-b11b-abd7f02b03cc","Type":"ContainerDied","Data":"ceb6a7d5dd4b68c77c1d75049cb33adf9b2b3b2c991b223a628334d3f763dd97"}
Nov 03 09:00:15.235468 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:15.235453    4473 scope.go:122] "RemoveContainer" containerID="a1a52214f11364e349169c29e8b4f15aeabad1a4fca8c444f81e5b16ca5c8f78"
Nov 03 09:00:15.235747 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:15.235732    4473 scope.go:122] "RemoveContainer" containerID="ceb6a7d5dd4b68c77c1d75049cb33adf9b2b3b2c991b223a628334d3f763dd97"
Nov 03 09:00:15.236046 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:00:15.236024    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-18f9da-29894940-phzwc_odh-ai-gateway-infra(b090afa5-9f65-4d79-b11b-abd7f02b03cc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894940-phzwc" podUID="b090afa5-9f65-4d79-b11b-abd7f02b03cc"
Nov 03 09:00:15.237497 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:15.237477    4473 generic.go:356] "Generic (PLEG): container finished" podID="2967ad42-d177-44c0-bdbb-27818c7a44cf" containerID="4cef6b493ad5f91903650ff52b0d817ccf42ee5dfe97b3a499a304645ab7efa3" exitCode=60
Nov 03 09:00:15.237579 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:15.237504    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894940-2wdhm" event={"ID":"2967ad42-d177-44c0-bdbb-27818c7a44cf","Type":"ContainerDied","Data":"4cef6b493ad5f91903650ff52b0d817ccf42ee5dfe97b3a499a304645ab7efa3"}
Nov 03 09:00:15.237788 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:15.237777    4473 scope.go:122] "RemoveContainer" containerID="4cef6b493ad5f91903650ff52b0d817ccf42ee5dfe97b3a499a304645ab7efa3"
Nov 03 09:00:15.238009 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:00:15.237993    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-910d50-29894940-2wdhm_odh-ai-gateway-infra(2967ad42-d177-44c0-bdbb-27818c7a44cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894940-2wdhm" podUID="2967ad42-d177-44c0-bdbb-27818c7a44cf"
Nov 03 09:00:15.247174 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:15.247149    4473 scope.go:122] "RemoveContainer" containerID="2b757546955b05bb4f26a62b8caf0bcd4c778b5f6a84cdec5bb2644eb38af3d4"
Nov 03 09:00:15.351116 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:15.351067    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894940-gzdxr" podStartSLOduration=15.351049662 podStartE2EDuration="15.351049662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:00:15.348256972 +0000 UTC m=+1239.617320431" watchObservedRunningTime="2026-11-03 09:00:15.351049662 +0000 UTC m=+1239.620113095"
Nov 03 09:00:15.357741 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:15.357716    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894940-phzwc"]
Nov 03 09:00:15.473260 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:15.473186    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894940-2wdhm"]
Nov 03 09:00:16.249988 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:16.249921    4473 generic.go:356] "Generic (PLEG): container finished" podID="b7fc3bdf-392e-4042-a0af-aed703adc65c" containerID="c1583df594436e6ca094b070e66052dda8a406fcdc19e41d7ea2e9309bf2e1b6" exitCode=60
Nov 03 09:00:16.250474 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:16.249993    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894940-gzdxr" event={"ID":"b7fc3bdf-392e-4042-a0af-aed703adc65c","Type":"ContainerDied","Data":"c1583df594436e6ca094b070e66052dda8a406fcdc19e41d7ea2e9309bf2e1b6"}
Nov 03 09:00:16.250603 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:16.250552    4473 scope.go:122] "RemoveContainer" containerID="9e2fa84fdc0f01b94c274d8bc4e11211aee8c9c06587d08131e1f6e5cdc208f0"
Nov 03 09:00:16.250977 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:16.250939    4473 scope.go:122] "RemoveContainer" containerID="c1583df594436e6ca094b070e66052dda8a406fcdc19e41d7ea2e9309bf2e1b6"
Nov 03 09:00:16.251272 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:00:16.251250    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-77207e-29894940-gzdxr_odh-ai-gateway-infra(b7fc3bdf-392e-4042-a0af-aed703adc65c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894940-gzdxr" podUID="b7fc3bdf-392e-4042-a0af-aed703adc65c"
Nov 03 09:00:16.362765 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:16.362731    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894940-gzdxr"]
Nov 03 09:00:16.415768 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:16.415742    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894940-2wdhm"
Nov 03 09:00:16.450739 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:16.450719    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894940-phzwc"
Nov 03 09:00:16.465621 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:16.465601    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b090afa5-9f65-4d79-b11b-abd7f02b03cc-kube-api-access-zdpdp\" (UniqueName: \"kubernetes.io/projected/b090afa5-9f65-4d79-b11b-abd7f02b03cc-kube-api-access-zdpdp\") pod \"b090afa5-9f65-4d79-b11b-abd7f02b03cc\" (UID: \"b090afa5-9f65-4d79-b11b-abd7f02b03cc\") "
Nov 03 09:00:16.465696 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:16.465664    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2967ad42-d177-44c0-bdbb-27818c7a44cf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2967ad42-d177-44c0-bdbb-27818c7a44cf-maas-api-service-ca\") pod \"2967ad42-d177-44c0-bdbb-27818c7a44cf\" (UID: \"2967ad42-d177-44c0-bdbb-27818c7a44cf\") "
Nov 03 09:00:16.465743 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:16.465696    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b090afa5-9f65-4d79-b11b-abd7f02b03cc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b090afa5-9f65-4d79-b11b-abd7f02b03cc-maas-api-service-ca\") pod \"b090afa5-9f65-4d79-b11b-abd7f02b03cc\" (UID: \"b090afa5-9f65-4d79-b11b-abd7f02b03cc\") "
Nov 03 09:00:16.465743 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:16.465716    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2967ad42-d177-44c0-bdbb-27818c7a44cf-kube-api-access-k8sbb\" (UniqueName: \"kubernetes.io/projected/2967ad42-d177-44c0-bdbb-27818c7a44cf-kube-api-access-k8sbb\") pod \"2967ad42-d177-44c0-bdbb-27818c7a44cf\" (UID: \"2967ad42-d177-44c0-bdbb-27818c7a44cf\") "
Nov 03 09:00:16.466063 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:16.466040    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b090afa5-9f65-4d79-b11b-abd7f02b03cc-maas-api-service-ca" pod "b090afa5-9f65-4d79-b11b-abd7f02b03cc" (UID: "b090afa5-9f65-4d79-b11b-abd7f02b03cc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:00:16.466151 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:16.466079    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2967ad42-d177-44c0-bdbb-27818c7a44cf-maas-api-service-ca" pod "2967ad42-d177-44c0-bdbb-27818c7a44cf" (UID: "2967ad42-d177-44c0-bdbb-27818c7a44cf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:00:16.467842 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:16.467818    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b090afa5-9f65-4d79-b11b-abd7f02b03cc-kube-api-access-zdpdp" pod "b090afa5-9f65-4d79-b11b-abd7f02b03cc" (UID: "b090afa5-9f65-4d79-b11b-abd7f02b03cc"). InnerVolumeSpecName "kube-api-access-zdpdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:00:16.467935 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:16.467849    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2967ad42-d177-44c0-bdbb-27818c7a44cf-kube-api-access-k8sbb" pod "2967ad42-d177-44c0-bdbb-27818c7a44cf" (UID: "2967ad42-d177-44c0-bdbb-27818c7a44cf"). InnerVolumeSpecName "kube-api-access-k8sbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:00:16.567571 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:16.567484    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2967ad42-d177-44c0-bdbb-27818c7a44cf-maas-api-service-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:00:16.567571 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:16.567514    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b090afa5-9f65-4d79-b11b-abd7f02b03cc-maas-api-service-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:00:16.567571 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:16.567528    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8sbb\" (UniqueName: \"kubernetes.io/projected/2967ad42-d177-44c0-bdbb-27818c7a44cf-kube-api-access-k8sbb\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:00:16.567571 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:16.567540    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdpdp\" (UniqueName: \"kubernetes.io/projected/b090afa5-9f65-4d79-b11b-abd7f02b03cc-kube-api-access-zdpdp\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:00:16.937213 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:00:16.937179    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:00:17.257157 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:17.257075    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894940-phzwc"
Nov 03 09:00:17.257157 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:17.257116    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894940-phzwc" event={"ID":"b090afa5-9f65-4d79-b11b-abd7f02b03cc","Type":"ContainerDied","Data":"bea4b6c861cf67d2f6a4394fd5894bc1b86167d90d54cdc7bafc4afa37336e48"}
Nov 03 09:00:17.257639 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:17.257182    4473 scope.go:122] "RemoveContainer" containerID="ceb6a7d5dd4b68c77c1d75049cb33adf9b2b3b2c991b223a628334d3f763dd97"
Nov 03 09:00:17.258789 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:17.258761    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894940-2wdhm"
Nov 03 09:00:17.258789 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:17.258782    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894940-2wdhm" event={"ID":"2967ad42-d177-44c0-bdbb-27818c7a44cf","Type":"ContainerDied","Data":"5c9f1632c232be18d27a7e5c3ebfedc20bfa6fcab4cb70415fe5eb6b60e5bf72"}
Nov 03 09:00:17.269432 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:17.269413    4473 scope.go:122] "RemoveContainer" containerID="4cef6b493ad5f91903650ff52b0d817ccf42ee5dfe97b3a499a304645ab7efa3"
Nov 03 09:00:17.410241 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:17.410214    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894940-gzdxr"
Nov 03 09:00:17.426399 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:17.426370    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894940-2wdhm"]
Nov 03 09:00:17.433713 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:17.433690    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894940-2wdhm"]
Nov 03 09:00:17.478465 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:17.478439    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7fc3bdf-392e-4042-a0af-aed703adc65c-kube-api-access-bfglf\" (UniqueName: \"kubernetes.io/projected/b7fc3bdf-392e-4042-a0af-aed703adc65c-kube-api-access-bfglf\") pod \"b7fc3bdf-392e-4042-a0af-aed703adc65c\" (UID: \"b7fc3bdf-392e-4042-a0af-aed703adc65c\") "
Nov 03 09:00:17.478582 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:17.478569    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7fc3bdf-392e-4042-a0af-aed703adc65c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7fc3bdf-392e-4042-a0af-aed703adc65c-maas-api-service-ca\") pod \"b7fc3bdf-392e-4042-a0af-aed703adc65c\" (UID: \"b7fc3bdf-392e-4042-a0af-aed703adc65c\") "
Nov 03 09:00:17.478882 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:17.478862    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7fc3bdf-392e-4042-a0af-aed703adc65c-maas-api-service-ca" pod "b7fc3bdf-392e-4042-a0af-aed703adc65c" (UID: "b7fc3bdf-392e-4042-a0af-aed703adc65c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:00:17.480597 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:17.480578    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7fc3bdf-392e-4042-a0af-aed703adc65c-kube-api-access-bfglf" pod "b7fc3bdf-392e-4042-a0af-aed703adc65c" (UID: "b7fc3bdf-392e-4042-a0af-aed703adc65c"). InnerVolumeSpecName "kube-api-access-bfglf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:00:17.545843 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:17.545780    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894940-phzwc"]
Nov 03 09:00:17.552505 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:17.552484    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894940-phzwc"]
Nov 03 09:00:17.579858 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:17.579833    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7fc3bdf-392e-4042-a0af-aed703adc65c-maas-api-service-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:00:17.579858 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:17.579854    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfglf\" (UniqueName: \"kubernetes.io/projected/b7fc3bdf-392e-4042-a0af-aed703adc65c-kube-api-access-bfglf\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:00:17.870124 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:17.870092    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2967ad42-d177-44c0-bdbb-27818c7a44cf" path="/var/lib/kubelet/pods/2967ad42-d177-44c0-bdbb-27818c7a44cf/volumes"
Nov 03 09:00:17.870641 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:17.870625    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b090afa5-9f65-4d79-b11b-abd7f02b03cc" path="/var/lib/kubelet/pods/b090afa5-9f65-4d79-b11b-abd7f02b03cc/volumes"
Nov 03 09:00:18.267241 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:18.267166    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894940-gzdxr"
Nov 03 09:00:18.267241 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:18.267182    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894940-gzdxr" event={"ID":"b7fc3bdf-392e-4042-a0af-aed703adc65c","Type":"ContainerDied","Data":"c29c6b59e98db482047f647087865fd826a2b35a86cef2128a46601f95efd8de"}
Nov 03 09:00:18.267779 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:18.267252    4473 scope.go:122] "RemoveContainer" containerID="c1583df594436e6ca094b070e66052dda8a406fcdc19e41d7ea2e9309bf2e1b6"
Nov 03 09:00:18.436363 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:18.436332    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894940-gzdxr"]
Nov 03 09:00:18.441758 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:18.441732    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894940-gzdxr"]
Nov 03 09:00:19.869435 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:19.869390    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7fc3bdf-392e-4042-a0af-aed703adc65c" path="/var/lib/kubelet/pods/b7fc3bdf-392e-4042-a0af-aed703adc65c/volumes"
Nov 03 09:00:21.098644 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:21.098615    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-910d50/e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf"]
Nov 03 09:00:26.053632 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.053598    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-689cb311-wkgx9"]
Nov 03 09:00:26.103386 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.103333    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-689cb311-wkgx9"]
Nov 03 09:00:26.477077 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.477038    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"]
Nov 03 09:00:26.477715 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.477692    4473 cpu_manager.go:414] "containerMap: removing container" podUID="b7fc3bdf-392e-4042-a0af-aed703adc65c" containerName="cleanup"
Nov 03 09:00:26.477715 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.477713    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7fc3bdf-392e-4042-a0af-aed703adc65c" containerName="cleanup"
Nov 03 09:00:26.477909 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.477738    4473 cpu_manager.go:414] "containerMap: removing container" podUID="2967ad42-d177-44c0-bdbb-27818c7a44cf" containerName="cleanup"
Nov 03 09:00:26.477909 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.477745    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2967ad42-d177-44c0-bdbb-27818c7a44cf" containerName="cleanup"
Nov 03 09:00:26.477909 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.477766    4473 cpu_manager.go:414] "containerMap: removing container" podUID="b7fc3bdf-392e-4042-a0af-aed703adc65c" containerName="cleanup"
Nov 03 09:00:26.477909 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.477771    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7fc3bdf-392e-4042-a0af-aed703adc65c" containerName="cleanup"
Nov 03 09:00:26.477909 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.477778    4473 cpu_manager.go:414] "containerMap: removing container" podUID="b090afa5-9f65-4d79-b11b-abd7f02b03cc" containerName="cleanup"
Nov 03 09:00:26.477909 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.477782    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b090afa5-9f65-4d79-b11b-abd7f02b03cc" containerName="cleanup"
Nov 03 09:00:26.477909 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.477789    4473 cpu_manager.go:414] "containerMap: removing container" podUID="b7fc3bdf-392e-4042-a0af-aed703adc65c" containerName="cleanup"
Nov 03 09:00:26.477909 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.477793    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7fc3bdf-392e-4042-a0af-aed703adc65c" containerName="cleanup"
Nov 03 09:00:26.477909 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.477802    4473 cpu_manager.go:414] "containerMap: removing container" podUID="2967ad42-d177-44c0-bdbb-27818c7a44cf" containerName="cleanup"
Nov 03 09:00:26.477909 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.477806    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2967ad42-d177-44c0-bdbb-27818c7a44cf" containerName="cleanup"
Nov 03 09:00:26.477909 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.477828    4473 cpu_manager.go:414] "containerMap: removing container" podUID="b090afa5-9f65-4d79-b11b-abd7f02b03cc" containerName="cleanup"
Nov 03 09:00:26.477909 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.477832    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b090afa5-9f65-4d79-b11b-abd7f02b03cc" containerName="cleanup"
Nov 03 09:00:26.477909 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.477901    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7fc3bdf-392e-4042-a0af-aed703adc65c" containerName="cleanup"
Nov 03 09:00:26.477909 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.477907    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b090afa5-9f65-4d79-b11b-abd7f02b03cc" containerName="cleanup"
Nov 03 09:00:26.477909 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.477913    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="2967ad42-d177-44c0-bdbb-27818c7a44cf" containerName="cleanup"
Nov 03 09:00:26.477909 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.477919    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7fc3bdf-392e-4042-a0af-aed703adc65c" containerName="cleanup"
Nov 03 09:00:26.478745 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.477928    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b090afa5-9f65-4d79-b11b-abd7f02b03cc" containerName="cleanup"
Nov 03 09:00:26.478745 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.477934    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="2967ad42-d177-44c0-bdbb-27818c7a44cf" containerName="cleanup"
Nov 03 09:00:26.481093 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.481077    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"
Nov 03 09:00:26.540387 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.540359    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-26e861-openshift-default-dockercfg-4wl7s\""
Nov 03 09:00:26.543163 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.543136    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"]
Nov 03 09:00:26.579100 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.579070    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb6f0603-0fa2-4223-b0d6-5411752e25cc-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb\" (UID: \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"
Nov 03 09:00:26.579272 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.579124    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb6f0603-0fa2-4223-b0d6-5411752e25cc-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb\" (UID: \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"
Nov 03 09:00:26.579272 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.579216    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb6f0603-0fa2-4223-b0d6-5411752e25cc-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb\" (UID: \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"
Nov 03 09:00:26.579356 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.579277    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxpd7\" (UniqueName: \"kubernetes.io/projected/cb6f0603-0fa2-4223-b0d6-5411752e25cc-kube-api-access-zxpd7\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb\" (UID: \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"
Nov 03 09:00:26.579419 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.579401    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb6f0603-0fa2-4223-b0d6-5411752e25cc-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb\" (UID: \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"
Nov 03 09:00:26.579461 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.579447    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb6f0603-0fa2-4223-b0d6-5411752e25cc-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb\" (UID: \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"
Nov 03 09:00:26.579504 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.579489    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb6f0603-0fa2-4223-b0d6-5411752e25cc-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb\" (UID: \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"
Nov 03 09:00:26.579554 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.579516    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb6f0603-0fa2-4223-b0d6-5411752e25cc-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb\" (UID: \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"
Nov 03 09:00:26.579612 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.579598    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb6f0603-0fa2-4223-b0d6-5411752e25cc-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb\" (UID: \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"
Nov 03 09:00:26.681093 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.681060    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb6f0603-0fa2-4223-b0d6-5411752e25cc-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb\" (UID: \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"
Nov 03 09:00:26.681260 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.681109    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb6f0603-0fa2-4223-b0d6-5411752e25cc-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb\" (UID: \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"
Nov 03 09:00:26.681260 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.681139    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb6f0603-0fa2-4223-b0d6-5411752e25cc-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb\" (UID: \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"
Nov 03 09:00:26.681260 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.681169    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb6f0603-0fa2-4223-b0d6-5411752e25cc-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb\" (UID: \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"
Nov 03 09:00:26.681260 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.681207    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb6f0603-0fa2-4223-b0d6-5411752e25cc-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb\" (UID: \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"
Nov 03 09:00:26.681260 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.681255    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb6f0603-0fa2-4223-b0d6-5411752e25cc-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb\" (UID: \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"
Nov 03 09:00:26.681519 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.681291    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb6f0603-0fa2-4223-b0d6-5411752e25cc-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb\" (UID: \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"
Nov 03 09:00:26.681519 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.681324    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb6f0603-0fa2-4223-b0d6-5411752e25cc-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb\" (UID: \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"
Nov 03 09:00:26.681519 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.681366    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxpd7\" (UniqueName: \"kubernetes.io/projected/cb6f0603-0fa2-4223-b0d6-5411752e25cc-kube-api-access-zxpd7\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb\" (UID: \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"
Nov 03 09:00:26.681715 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.681683    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb6f0603-0fa2-4223-b0d6-5411752e25cc-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb\" (UID: \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"
Nov 03 09:00:26.681789 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.681737    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb6f0603-0fa2-4223-b0d6-5411752e25cc-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb\" (UID: \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"
Nov 03 09:00:26.681852 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.681814    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb6f0603-0fa2-4223-b0d6-5411752e25cc-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb\" (UID: \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"
Nov 03 09:00:26.681911 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.681896    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb6f0603-0fa2-4223-b0d6-5411752e25cc-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb\" (UID: \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"
Nov 03 09:00:26.681990 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.681938    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb6f0603-0fa2-4223-b0d6-5411752e25cc-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb\" (UID: \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"
Nov 03 09:00:26.683565 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.683547    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb6f0603-0fa2-4223-b0d6-5411752e25cc-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb\" (UID: \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"
Nov 03 09:00:26.683831 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.683809    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb6f0603-0fa2-4223-b0d6-5411752e25cc-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb\" (UID: \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"
Nov 03 09:00:26.740606 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.740541    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb6f0603-0fa2-4223-b0d6-5411752e25cc-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb\" (UID: \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"
Nov 03 09:00:26.740915 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.740895    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxpd7\" (UniqueName: \"kubernetes.io/projected/cb6f0603-0fa2-4223-b0d6-5411752e25cc-kube-api-access-zxpd7\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb\" (UID: \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"
Nov 03 09:00:26.794526 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:26.794499    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"
Nov 03 09:00:27.230837 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:27.230811    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"]
Nov 03 09:00:27.232098 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:00:27.232076    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb6f0603_0fa2_4223_b0d6_5411752e25cc.slice/crio-766048bd23ab0bb113f201b6074a96726160fd8d8f997b96d89c022d1bd34dda WatchSource:0}: Error finding container 766048bd23ab0bb113f201b6074a96726160fd8d8f997b96d89c022d1bd34dda: Status 404 returned error can't find the container with id 766048bd23ab0bb113f201b6074a96726160fd8d8f997b96d89c022d1bd34dda
Nov 03 09:00:27.234412 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:27.234383    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:00:27.234486 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:27.234443    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:00:27.234486 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:27.234472    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:00:27.327502 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:27.327468    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb" event={"ID":"cb6f0603-0fa2-4223-b0d6-5411752e25cc","Type":"ContainerStarted","Data":"fbe4570e2c773b2572cdea384003f88d8058bea1988fcb8201480649614e3bc7"}
Nov 03 09:00:27.327684 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:27.327529    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb" event={"ID":"cb6f0603-0fa2-4223-b0d6-5411752e25cc","Type":"ContainerStarted","Data":"766048bd23ab0bb113f201b6074a96726160fd8d8f997b96d89c022d1bd34dda"}
Nov 03 09:00:27.446675 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:27.446612    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb" podStartSLOduration=1.446596262 podStartE2EDuration="1.446596262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:00:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:00:27.445298558 +0000 UTC m=+1251.714361992" watchObservedRunningTime="2026-11-03 09:00:27.446596262 +0000 UTC m=+1251.715659699"
Nov 03 09:00:27.795202 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:27.795174    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"
Nov 03 09:00:27.870116 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:27.870081    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61c2cf80-4442-4226-8b29-ff9ec8245461" path="/var/lib/kubelet/pods/61c2cf80-4442-4226-8b29-ff9ec8245461/volumes"
Nov 03 09:00:28.796892 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:28.796851    4473 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 09:00:28.797408 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:28.796902    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb" podUID="cb6f0603-0fa2-4223-b0d6-5411752e25cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 09:00:29.795075 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:29.795045    4473 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 09:00:29.795270 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:29.795096    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb" podUID="cb6f0603-0fa2-4223-b0d6-5411752e25cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 09:00:29.993159 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:29.993131    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70c3db8c-r64gq"]
Nov 03 09:00:29.993604 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:29.993590    4473 cpu_manager.go:414] "containerMap: removing container" podUID="b090afa5-9f65-4d79-b11b-abd7f02b03cc" containerName="cleanup"
Nov 03 09:00:29.993644 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:29.993605    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b090afa5-9f65-4d79-b11b-abd7f02b03cc" containerName="cleanup"
Nov 03 09:00:29.993644 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:29.993618    4473 cpu_manager.go:414] "containerMap: removing container" podUID="2967ad42-d177-44c0-bdbb-27818c7a44cf" containerName="cleanup"
Nov 03 09:00:29.993644 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:29.993623    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2967ad42-d177-44c0-bdbb-27818c7a44cf" containerName="cleanup"
Nov 03 09:00:29.993734 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:29.993695    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7fc3bdf-392e-4042-a0af-aed703adc65c" containerName="cleanup"
Nov 03 09:00:29.993734 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:29.993707    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b090afa5-9f65-4d79-b11b-abd7f02b03cc" containerName="cleanup"
Nov 03 09:00:29.993734 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:29.993714    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="2967ad42-d177-44c0-bdbb-27818c7a44cf" containerName="cleanup"
Nov 03 09:00:29.997082 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:29.997065    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c3db8c-r64gq"
Nov 03 09:00:30.054544 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:30.054468    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70c3db8c-r64gq"]
Nov 03 09:00:30.101764 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:30.101719    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:00:30.101764 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:30.101737    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:00:30.101764 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:30.101775    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:00:30.102023 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:30.101848    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:00:30.143689 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:30.143654    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwfbq\" (UniqueName: \"kubernetes.io/projected/9abf6bd6-23a7-4398-92ab-27c1d16ac7f9-kube-api-access-lwfbq\") pod \"maas-api-revoke-sub-70c3db8c-r64gq\" (UID: \"9abf6bd6-23a7-4398-92ab-27c1d16ac7f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c3db8c-r64gq"
Nov 03 09:00:30.143857 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:30.143733    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9abf6bd6-23a7-4398-92ab-27c1d16ac7f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70c3db8c-r64gq\" (UID: \"9abf6bd6-23a7-4398-92ab-27c1d16ac7f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c3db8c-r64gq"
Nov 03 09:00:30.143938 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:30.143920    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9abf6bd6-23a7-4398-92ab-27c1d16ac7f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70c3db8c-r64gq\" (UID: \"9abf6bd6-23a7-4398-92ab-27c1d16ac7f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c3db8c-r64gq"
Nov 03 09:00:30.244805 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:30.244760    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9abf6bd6-23a7-4398-92ab-27c1d16ac7f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70c3db8c-r64gq\" (UID: \"9abf6bd6-23a7-4398-92ab-27c1d16ac7f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c3db8c-r64gq"
Nov 03 09:00:30.245049 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:30.244862    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwfbq\" (UniqueName: \"kubernetes.io/projected/9abf6bd6-23a7-4398-92ab-27c1d16ac7f9-kube-api-access-lwfbq\") pod \"maas-api-revoke-sub-70c3db8c-r64gq\" (UID: \"9abf6bd6-23a7-4398-92ab-27c1d16ac7f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c3db8c-r64gq"
Nov 03 09:00:30.245049 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:30.244959    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9abf6bd6-23a7-4398-92ab-27c1d16ac7f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70c3db8c-r64gq\" (UID: \"9abf6bd6-23a7-4398-92ab-27c1d16ac7f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c3db8c-r64gq"
Nov 03 09:00:30.245498 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:30.245470    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9abf6bd6-23a7-4398-92ab-27c1d16ac7f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70c3db8c-r64gq\" (UID: \"9abf6bd6-23a7-4398-92ab-27c1d16ac7f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c3db8c-r64gq"
Nov 03 09:00:30.247460 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:30.247441    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9abf6bd6-23a7-4398-92ab-27c1d16ac7f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70c3db8c-r64gq\" (UID: \"9abf6bd6-23a7-4398-92ab-27c1d16ac7f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c3db8c-r64gq"
Nov 03 09:00:30.304290 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:30.304261    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwfbq\" (UniqueName: \"kubernetes.io/projected/9abf6bd6-23a7-4398-92ab-27c1d16ac7f9-kube-api-access-lwfbq\") pod \"maas-api-revoke-sub-70c3db8c-r64gq\" (UID: \"9abf6bd6-23a7-4398-92ab-27c1d16ac7f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c3db8c-r64gq"
Nov 03 09:00:30.310190 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:30.310139    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c3db8c-r64gq"
Nov 03 09:00:30.736969 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:30.717510    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70c3db8c-r64gq"]
Nov 03 09:00:30.742974 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:00:30.742917    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9abf6bd6_23a7_4398_92ab_27c1d16ac7f9.slice/crio-86a24e992a58c5dffd592413ecbe49d901786b90c2431762fa52bee36ecc2f12 WatchSource:0}: Error finding container 86a24e992a58c5dffd592413ecbe49d901786b90c2431762fa52bee36ecc2f12: Status 404 returned error can't find the container with id 86a24e992a58c5dffd592413ecbe49d901786b90c2431762fa52bee36ecc2f12
Nov 03 09:00:30.795026 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:30.794997    4473 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 09:00:30.795114 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:30.795047    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb" podUID="cb6f0603-0fa2-4223-b0d6-5411752e25cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 09:00:31.359743 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:31.359709    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c3db8c-r64gq" event={"ID":"9abf6bd6-23a7-4398-92ab-27c1d16ac7f9","Type":"ContainerStarted","Data":"0e8ade94451328b89152a3d2226b90a71bf2ac2feeee333da233c1c801015404"}
Nov 03 09:00:31.360228 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:31.359760    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c3db8c-r64gq" event={"ID":"9abf6bd6-23a7-4398-92ab-27c1d16ac7f9","Type":"ContainerStarted","Data":"86a24e992a58c5dffd592413ecbe49d901786b90c2431762fa52bee36ecc2f12"}
Nov 03 09:00:31.476891 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:31.476793    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c3db8c-r64gq" podStartSLOduration=2.476778675 podStartE2EDuration="2.476778675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:00:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:00:31.475024895 +0000 UTC m=+1255.744088330" watchObservedRunningTime="2026-11-03 09:00:31.476778675 +0000 UTC m=+1255.745842161"
Nov 03 09:00:31.795703 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:31.795679    4473 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 09:00:31.795799 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:31.795731    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb" podUID="cb6f0603-0fa2-4223-b0d6-5411752e25cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 09:00:32.366607 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:32.366569    4473 generic.go:356] "Generic (PLEG): container finished" podID="9abf6bd6-23a7-4398-92ab-27c1d16ac7f9" containerID="0e8ade94451328b89152a3d2226b90a71bf2ac2feeee333da233c1c801015404" exitCode=0
Nov 03 09:00:32.367024 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:32.366653    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c3db8c-r64gq" event={"ID":"9abf6bd6-23a7-4398-92ab-27c1d16ac7f9","Type":"ContainerDied","Data":"0e8ade94451328b89152a3d2226b90a71bf2ac2feeee333da233c1c801015404"}
Nov 03 09:00:32.795853 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:32.795765    4473 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 09:00:32.795853 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:32.795815    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb" podUID="cb6f0603-0fa2-4223-b0d6-5411752e25cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 09:00:33.518926 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:33.518897    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c3db8c-r64gq"
Nov 03 09:00:33.683663 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:33.683589    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9abf6bd6-23a7-4398-92ab-27c1d16ac7f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9abf6bd6-23a7-4398-92ab-27c1d16ac7f9-maas-api-openshift-ca\") pod \"9abf6bd6-23a7-4398-92ab-27c1d16ac7f9\" (UID: \"9abf6bd6-23a7-4398-92ab-27c1d16ac7f9\") "
Nov 03 09:00:33.683798 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:33.683775    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9abf6bd6-23a7-4398-92ab-27c1d16ac7f9-kube-api-access-lwfbq\" (UniqueName: \"kubernetes.io/projected/9abf6bd6-23a7-4398-92ab-27c1d16ac7f9-kube-api-access-lwfbq\") pod \"9abf6bd6-23a7-4398-92ab-27c1d16ac7f9\" (UID: \"9abf6bd6-23a7-4398-92ab-27c1d16ac7f9\") "
Nov 03 09:00:33.683859 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:33.683843    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9abf6bd6-23a7-4398-92ab-27c1d16ac7f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9abf6bd6-23a7-4398-92ab-27c1d16ac7f9-maas-api-xks-ca\") pod \"9abf6bd6-23a7-4398-92ab-27c1d16ac7f9\" (UID: \"9abf6bd6-23a7-4398-92ab-27c1d16ac7f9\") "
Nov 03 09:00:33.684031 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:33.684009    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9abf6bd6-23a7-4398-92ab-27c1d16ac7f9-maas-api-openshift-ca" pod "9abf6bd6-23a7-4398-92ab-27c1d16ac7f9" (UID: "9abf6bd6-23a7-4398-92ab-27c1d16ac7f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:00:33.684228 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:33.684204    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9abf6bd6-23a7-4398-92ab-27c1d16ac7f9-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:00:33.686110 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:33.686082    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9abf6bd6-23a7-4398-92ab-27c1d16ac7f9-maas-api-xks-ca" pod "9abf6bd6-23a7-4398-92ab-27c1d16ac7f9" (UID: "9abf6bd6-23a7-4398-92ab-27c1d16ac7f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:00:33.686197 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:33.686120    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9abf6bd6-23a7-4398-92ab-27c1d16ac7f9-kube-api-access-lwfbq" pod "9abf6bd6-23a7-4398-92ab-27c1d16ac7f9" (UID: "9abf6bd6-23a7-4398-92ab-27c1d16ac7f9"). InnerVolumeSpecName "kube-api-access-lwfbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:00:33.785445 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:33.785420    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwfbq\" (UniqueName: \"kubernetes.io/projected/9abf6bd6-23a7-4398-92ab-27c1d16ac7f9-kube-api-access-lwfbq\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:00:33.785599 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:33.785455    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9abf6bd6-23a7-4398-92ab-27c1d16ac7f9-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:00:33.795745 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:33.795724    4473 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 09:00:33.795824 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:33.795772    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb" podUID="cb6f0603-0fa2-4223-b0d6-5411752e25cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 09:00:34.382690 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:34.382659    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c3db8c-r64gq"
Nov 03 09:00:34.382690 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:34.382681    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70c3db8c-r64gq" event={"ID":"9abf6bd6-23a7-4398-92ab-27c1d16ac7f9","Type":"ContainerDied","Data":"86a24e992a58c5dffd592413ecbe49d901786b90c2431762fa52bee36ecc2f12"}
Nov 03 09:00:34.382917 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:34.382729    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86a24e992a58c5dffd592413ecbe49d901786b90c2431762fa52bee36ecc2f12"
Nov 03 09:00:34.799537 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:34.799462    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"
Nov 03 09:00:34.799890 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:34.799732    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"
Nov 03 09:00:34.800571 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:34.800551    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"
Nov 03 09:00:34.949279 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:34.949248    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:00:36.121792 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:36.121759    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf" podUID="d26a0519-b53f-4b0a-881d-1bcb2e10b8d5" containerName="main" containerID="cri-o://5f5560fd4725dab1b9175d43ae43590103745ab1bc14673883828f02a66fde76" gracePeriod=45
Nov 03 09:00:36.398626 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:36.398594    4473 generic.go:356] "Generic (PLEG): container finished" podID="d26a0519-b53f-4b0a-881d-1bcb2e10b8d5" containerID="5f5560fd4725dab1b9175d43ae43590103745ab1bc14673883828f02a66fde76" exitCode=0
Nov 03 09:00:36.398779 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:36.398677    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf" event={"ID":"d26a0519-b53f-4b0a-881d-1bcb2e10b8d5","Type":"ContainerDied","Data":"5f5560fd4725dab1b9175d43ae43590103745ab1bc14673883828f02a66fde76"}
Nov 03 09:00:36.398779 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:36.398731    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf" event={"ID":"d26a0519-b53f-4b0a-881d-1bcb2e10b8d5","Type":"ContainerDied","Data":"0ff11887de177980b9411256a45718abfd2f67c892caba5a09b4c53754cf580e"}
Nov 03 09:00:36.398779 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:36.398750    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ff11887de177980b9411256a45718abfd2f67c892caba5a09b4c53754cf580e"
Nov 03 09:00:36.406097 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:36.406076    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf"
Nov 03 09:00:36.549737 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:36.548334    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-model-cache\") pod \"d26a0519-b53f-4b0a-881d-1bcb2e10b8d5\" (UID: \"d26a0519-b53f-4b0a-881d-1bcb2e10b8d5\") "
Nov 03 09:00:36.549737 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:36.548392    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-dshm\") pod \"d26a0519-b53f-4b0a-881d-1bcb2e10b8d5\" (UID: \"d26a0519-b53f-4b0a-881d-1bcb2e10b8d5\") "
Nov 03 09:00:36.549737 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:36.548456    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-tmp-dir\") pod \"d26a0519-b53f-4b0a-881d-1bcb2e10b8d5\" (UID: \"d26a0519-b53f-4b0a-881d-1bcb2e10b8d5\") "
Nov 03 09:00:36.549737 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:36.548486    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-tls-certs\" (UniqueName: \"kubernetes.io/secret/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-tls-certs\") pod \"d26a0519-b53f-4b0a-881d-1bcb2e10b8d5\" (UID: \"d26a0519-b53f-4b0a-881d-1bcb2e10b8d5\") "
Nov 03 09:00:36.549737 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:36.548521    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-kube-api-access-h5w5l\" (UniqueName: \"kubernetes.io/projected/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-kube-api-access-h5w5l\") pod \"d26a0519-b53f-4b0a-881d-1bcb2e10b8d5\" (UID: \"d26a0519-b53f-4b0a-881d-1bcb2e10b8d5\") "
Nov 03 09:00:36.549737 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:36.548690    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-home\" (UniqueName: \"kubernetes.io/empty-dir/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-home\") pod \"d26a0519-b53f-4b0a-881d-1bcb2e10b8d5\" (UID: \"d26a0519-b53f-4b0a-881d-1bcb2e10b8d5\") "
Nov 03 09:00:36.562700 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:36.562634    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-model-cache" pod "d26a0519-b53f-4b0a-881d-1bcb2e10b8d5" (UID: "d26a0519-b53f-4b0a-881d-1bcb2e10b8d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:00:36.576271 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:36.576233    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-home" pod "d26a0519-b53f-4b0a-881d-1bcb2e10b8d5" (UID: "d26a0519-b53f-4b0a-881d-1bcb2e10b8d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:00:36.576674 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:36.576648    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-tmp-dir" pod "d26a0519-b53f-4b0a-881d-1bcb2e10b8d5" (UID: "d26a0519-b53f-4b0a-881d-1bcb2e10b8d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:00:36.584276 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:36.583733    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-dshm" pod "d26a0519-b53f-4b0a-881d-1bcb2e10b8d5" (UID: "d26a0519-b53f-4b0a-881d-1bcb2e10b8d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:00:36.584276 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:36.583873    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-kube-api-access-h5w5l" pod "d26a0519-b53f-4b0a-881d-1bcb2e10b8d5" (UID: "d26a0519-b53f-4b0a-881d-1bcb2e10b8d5"). InnerVolumeSpecName "kube-api-access-h5w5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:00:36.585099 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:36.585076    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-tls-certs" pod "d26a0519-b53f-4b0a-881d-1bcb2e10b8d5" (UID: "d26a0519-b53f-4b0a-881d-1bcb2e10b8d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:00:36.650026 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:36.649928    4473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-tmp-dir\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:00:36.650026 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:36.649975    4473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-tls-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:00:36.650026 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:36.649989    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5w5l\" (UniqueName: \"kubernetes.io/projected/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-kube-api-access-h5w5l\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:00:36.650026 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:36.650002    4473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-home\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:00:36.650026 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:36.650016    4473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-model-cache\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:00:36.650026 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:36.650027    4473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5-dshm\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:00:37.404683 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:37.404652    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf"
Nov 03 09:00:37.573025 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:37.572991    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-910d50/e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf"]
Nov 03 09:00:37.623069 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:37.623037    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-910d50/e2e-conflict-b-54a1bbf9-kserve-7fd7767875-5w2jf"]
Nov 03 09:00:37.869495 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:37.869465    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d26a0519-b53f-4b0a-881d-1bcb2e10b8d5" path="/var/lib/kubelet/pods/d26a0519-b53f-4b0a-881d-1bcb2e10b8d5/volumes"
Nov 03 09:00:43.465734 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:43.465699    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49033a88-gst8z"]
Nov 03 09:00:43.466164 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:43.466149    4473 cpu_manager.go:414] "containerMap: removing container" podUID="d26a0519-b53f-4b0a-881d-1bcb2e10b8d5" containerName="main"
Nov 03 09:00:43.466208 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:43.466164    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d26a0519-b53f-4b0a-881d-1bcb2e10b8d5" containerName="main"
Nov 03 09:00:43.466208 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:43.466193    4473 cpu_manager.go:414] "containerMap: removing container" podUID="9abf6bd6-23a7-4398-92ab-27c1d16ac7f9" containerName="revoke-keys"
Nov 03 09:00:43.466208 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:43.466197    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9abf6bd6-23a7-4398-92ab-27c1d16ac7f9" containerName="revoke-keys"
Nov 03 09:00:43.466304 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:43.466257    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="9abf6bd6-23a7-4398-92ab-27c1d16ac7f9" containerName="revoke-keys"
Nov 03 09:00:43.466304 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:43.466268    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d26a0519-b53f-4b0a-881d-1bcb2e10b8d5" containerName="main"
Nov 03 09:00:43.470425 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:43.470404    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49033a88-gst8z"
Nov 03 09:00:43.526637 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:43.526599    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49033a88-gst8z"]
Nov 03 09:00:43.574219 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:43.574196    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:00:43.574219 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:43.574206    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:00:43.574376 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:43.574243    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:00:43.574376 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:43.574245    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:00:43.621742 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:43.621707    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49033a88-gst8z\" (UID: \"ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49033a88-gst8z"
Nov 03 09:00:43.621884 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:43.621823    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49033a88-gst8z\" (UID: \"ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49033a88-gst8z"
Nov 03 09:00:43.621968 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:43.621932    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms4sf\" (UniqueName: \"kubernetes.io/projected/ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74-kube-api-access-ms4sf\") pod \"maas-api-revoke-sub-49033a88-gst8z\" (UID: \"ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49033a88-gst8z"
Nov 03 09:00:43.723463 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:43.723385    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49033a88-gst8z\" (UID: \"ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49033a88-gst8z"
Nov 03 09:00:43.723463 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:43.723450    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49033a88-gst8z\" (UID: \"ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49033a88-gst8z"
Nov 03 09:00:43.723659 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:43.723506    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms4sf\" (UniqueName: \"kubernetes.io/projected/ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74-kube-api-access-ms4sf\") pod \"maas-api-revoke-sub-49033a88-gst8z\" (UID: \"ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49033a88-gst8z"
Nov 03 09:00:43.724157 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:43.724137    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49033a88-gst8z\" (UID: \"ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49033a88-gst8z"
Nov 03 09:00:43.725937 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:43.725920    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49033a88-gst8z\" (UID: \"ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49033a88-gst8z"
Nov 03 09:00:43.782886 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:43.782857    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms4sf\" (UniqueName: \"kubernetes.io/projected/ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74-kube-api-access-ms4sf\") pod \"maas-api-revoke-sub-49033a88-gst8z\" (UID: \"ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49033a88-gst8z"
Nov 03 09:00:43.783005 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:43.782977    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49033a88-gst8z"
Nov 03 09:00:44.156211 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:44.156186    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49033a88-gst8z"]
Nov 03 09:00:44.157810 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:00:44.157781    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad684a9f_e9eb_4c0c_b3e3_5471b3ebbd74.slice/crio-9defdb908f50f5293a4a6d894bbf5c3491cd27d07125df29346e8776f98c9e99 WatchSource:0}: Error finding container 9defdb908f50f5293a4a6d894bbf5c3491cd27d07125df29346e8776f98c9e99: Status 404 returned error can't find the container with id 9defdb908f50f5293a4a6d894bbf5c3491cd27d07125df29346e8776f98c9e99
Nov 03 09:00:44.443049 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:44.443017    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49033a88-gst8z" event={"ID":"ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74","Type":"ContainerStarted","Data":"058a1b8533506f0403f410a5009bd9ed90f9649800baf63c0ab9fec2dc1055c8"}
Nov 03 09:00:44.443139 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:44.443077    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49033a88-gst8z" event={"ID":"ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74","Type":"ContainerStarted","Data":"9defdb908f50f5293a4a6d894bbf5c3491cd27d07125df29346e8776f98c9e99"}
Nov 03 09:00:44.558419 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:44.558370    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49033a88-gst8z" podStartSLOduration=1.558357686 podStartE2EDuration="1.558357686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:00:44.557626338 +0000 UTC m=+1268.826689793" watchObservedRunningTime="2026-11-03 09:00:44.558357686 +0000 UTC m=+1268.827421119"
Nov 03 09:00:45.450805 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:45.450772    4473 generic.go:356] "Generic (PLEG): container finished" podID="ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74" containerID="058a1b8533506f0403f410a5009bd9ed90f9649800baf63c0ab9fec2dc1055c8" exitCode=0
Nov 03 09:00:45.451040 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:45.450846    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49033a88-gst8z" event={"ID":"ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74","Type":"ContainerDied","Data":"058a1b8533506f0403f410a5009bd9ed90f9649800baf63c0ab9fec2dc1055c8"}
Nov 03 09:00:46.604049 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:46.604021    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49033a88-gst8z"
Nov 03 09:00:46.757825 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:46.757746    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74-maas-api-openshift-ca\") pod \"ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74\" (UID: \"ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74\") "
Nov 03 09:00:46.757991 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:46.757842    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74-maas-api-xks-ca\") pod \"ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74\" (UID: \"ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74\") "
Nov 03 09:00:46.757991 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:46.757870    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74-kube-api-access-ms4sf\" (UniqueName: \"kubernetes.io/projected/ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74-kube-api-access-ms4sf\") pod \"ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74\" (UID: \"ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74\") "
Nov 03 09:00:46.758170 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:46.758144    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74-maas-api-openshift-ca" pod "ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74" (UID: "ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:00:46.760077 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:46.760052    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74-maas-api-xks-ca" pod "ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74" (UID: "ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:00:46.760204 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:46.760128    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74-kube-api-access-ms4sf" pod "ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74" (UID: "ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74"). InnerVolumeSpecName "kube-api-access-ms4sf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:00:46.859055 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:46.859027    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:00:46.859055 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:46.859050    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms4sf\" (UniqueName: \"kubernetes.io/projected/ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74-kube-api-access-ms4sf\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:00:46.859055 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:46.859061    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:00:46.937699 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:00:46.937663    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:00:47.462607 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:47.462582    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49033a88-gst8z"
Nov 03 09:00:47.462817 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:47.462612    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49033a88-gst8z" event={"ID":"ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74","Type":"ContainerDied","Data":"9defdb908f50f5293a4a6d894bbf5c3491cd27d07125df29346e8776f98c9e99"}
Nov 03 09:00:47.462817 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:47.462669    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9defdb908f50f5293a4a6d894bbf5c3491cd27d07125df29346e8776f98c9e99"
Nov 03 09:00:57.055714 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:57.055675    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27858e00-l4lc5"]
Nov 03 09:00:57.105915 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:57.105884    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27858e00-l4lc5"]
Nov 03 09:00:57.564740 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:57.564700    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d2fc1b5-mbssl"]
Nov 03 09:00:57.565267 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:57.565249    4473 cpu_manager.go:414] "containerMap: removing container" podUID="ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74" containerName="revoke-keys"
Nov 03 09:00:57.565387 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:57.565268    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74" containerName="revoke-keys"
Nov 03 09:00:57.565387 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:57.565369    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74" containerName="revoke-keys"
Nov 03 09:00:57.568214 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:57.568196    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2fc1b5-mbssl"
Nov 03 09:00:57.626775 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:57.626751    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d2fc1b5-mbssl"]
Nov 03 09:00:57.673886 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:57.673852    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:00:57.673886 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:57.673886    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:00:57.674105 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:57.674000    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:00:57.674105 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:57.674072    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:00:57.708299 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:57.708276    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/346fcfc5-c545-4443-aae5-f3945c68b8fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d2fc1b5-mbssl\" (UID: \"346fcfc5-c545-4443-aae5-f3945c68b8fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2fc1b5-mbssl"
Nov 03 09:00:57.708419 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:57.708346    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/346fcfc5-c545-4443-aae5-f3945c68b8fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d2fc1b5-mbssl\" (UID: \"346fcfc5-c545-4443-aae5-f3945c68b8fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2fc1b5-mbssl"
Nov 03 09:00:57.708489 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:57.708471    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqdhb\" (UniqueName: \"kubernetes.io/projected/346fcfc5-c545-4443-aae5-f3945c68b8fe-kube-api-access-cqdhb\") pod \"maas-api-revoke-sub-5d2fc1b5-mbssl\" (UID: \"346fcfc5-c545-4443-aae5-f3945c68b8fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2fc1b5-mbssl"
Nov 03 09:00:57.809787 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:57.809759    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/346fcfc5-c545-4443-aae5-f3945c68b8fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d2fc1b5-mbssl\" (UID: \"346fcfc5-c545-4443-aae5-f3945c68b8fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2fc1b5-mbssl"
Nov 03 09:00:57.809975 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:57.809831    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/346fcfc5-c545-4443-aae5-f3945c68b8fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d2fc1b5-mbssl\" (UID: \"346fcfc5-c545-4443-aae5-f3945c68b8fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2fc1b5-mbssl"
Nov 03 09:00:57.809975 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:57.809866    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqdhb\" (UniqueName: \"kubernetes.io/projected/346fcfc5-c545-4443-aae5-f3945c68b8fe-kube-api-access-cqdhb\") pod \"maas-api-revoke-sub-5d2fc1b5-mbssl\" (UID: \"346fcfc5-c545-4443-aae5-f3945c68b8fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2fc1b5-mbssl"
Nov 03 09:00:57.810541 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:57.810518    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/346fcfc5-c545-4443-aae5-f3945c68b8fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d2fc1b5-mbssl\" (UID: \"346fcfc5-c545-4443-aae5-f3945c68b8fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2fc1b5-mbssl"
Nov 03 09:00:57.812337 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:57.812316    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/346fcfc5-c545-4443-aae5-f3945c68b8fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d2fc1b5-mbssl\" (UID: \"346fcfc5-c545-4443-aae5-f3945c68b8fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2fc1b5-mbssl"
Nov 03 09:00:57.869642 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:57.869616    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bc4f54b-1dc8-4327-9d97-930ca5774ab9" path="/var/lib/kubelet/pods/5bc4f54b-1dc8-4327-9d97-930ca5774ab9/volumes"
Nov 03 09:00:57.871516 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:57.871487    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqdhb\" (UniqueName: \"kubernetes.io/projected/346fcfc5-c545-4443-aae5-f3945c68b8fe-kube-api-access-cqdhb\") pod \"maas-api-revoke-sub-5d2fc1b5-mbssl\" (UID: \"346fcfc5-c545-4443-aae5-f3945c68b8fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2fc1b5-mbssl"
Nov 03 09:00:57.881135 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:57.881118    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2fc1b5-mbssl"
Nov 03 09:00:58.257500 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:58.257470    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d2fc1b5-mbssl"]
Nov 03 09:00:58.258848 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:00:58.258822    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod346fcfc5_c545_4443_aae5_f3945c68b8fe.slice/crio-83c27cf7a2c58ff7c24b493bcf0d0f5151deb7e011c495f3cca64c3477a4d104 WatchSource:0}: Error finding container 83c27cf7a2c58ff7c24b493bcf0d0f5151deb7e011c495f3cca64c3477a4d104: Status 404 returned error can't find the container with id 83c27cf7a2c58ff7c24b493bcf0d0f5151deb7e011c495f3cca64c3477a4d104
Nov 03 09:00:58.527399 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:58.527372    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2fc1b5-mbssl" event={"ID":"346fcfc5-c545-4443-aae5-f3945c68b8fe","Type":"ContainerStarted","Data":"83c27cf7a2c58ff7c24b493bcf0d0f5151deb7e011c495f3cca64c3477a4d104"}
Nov 03 09:00:59.534854 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:59.534823    4473 generic.go:356] "Generic (PLEG): container finished" podID="346fcfc5-c545-4443-aae5-f3945c68b8fe" containerID="350f5f16eb6acbf239cd68905016614fb82cd7f980d4793bb56223a54b77ec40" exitCode=0
Nov 03 09:00:59.535342 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:00:59.534902    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2fc1b5-mbssl" event={"ID":"346fcfc5-c545-4443-aae5-f3945c68b8fe","Type":"ContainerDied","Data":"350f5f16eb6acbf239cd68905016614fb82cd7f980d4793bb56223a54b77ec40"}
Nov 03 09:01:00.683922 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:00.683898    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2fc1b5-mbssl"
Nov 03 09:01:00.841670 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:00.841598    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/346fcfc5-c545-4443-aae5-f3945c68b8fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/346fcfc5-c545-4443-aae5-f3945c68b8fe-maas-api-openshift-ca\") pod \"346fcfc5-c545-4443-aae5-f3945c68b8fe\" (UID: \"346fcfc5-c545-4443-aae5-f3945c68b8fe\") "
Nov 03 09:01:00.841670 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:00.841639    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/346fcfc5-c545-4443-aae5-f3945c68b8fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/346fcfc5-c545-4443-aae5-f3945c68b8fe-maas-api-xks-ca\") pod \"346fcfc5-c545-4443-aae5-f3945c68b8fe\" (UID: \"346fcfc5-c545-4443-aae5-f3945c68b8fe\") "
Nov 03 09:01:00.841833 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:00.841771    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/346fcfc5-c545-4443-aae5-f3945c68b8fe-kube-api-access-cqdhb\" (UniqueName: \"kubernetes.io/projected/346fcfc5-c545-4443-aae5-f3945c68b8fe-kube-api-access-cqdhb\") pod \"346fcfc5-c545-4443-aae5-f3945c68b8fe\" (UID: \"346fcfc5-c545-4443-aae5-f3945c68b8fe\") "
Nov 03 09:01:00.842055 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:00.842032    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/346fcfc5-c545-4443-aae5-f3945c68b8fe-maas-api-openshift-ca" pod "346fcfc5-c545-4443-aae5-f3945c68b8fe" (UID: "346fcfc5-c545-4443-aae5-f3945c68b8fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:01:00.842249 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:00.842234    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/346fcfc5-c545-4443-aae5-f3945c68b8fe-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:01:00.843938 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:00.843913    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/346fcfc5-c545-4443-aae5-f3945c68b8fe-maas-api-xks-ca" pod "346fcfc5-c545-4443-aae5-f3945c68b8fe" (UID: "346fcfc5-c545-4443-aae5-f3945c68b8fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:01:00.844133 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:00.844109    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/346fcfc5-c545-4443-aae5-f3945c68b8fe-kube-api-access-cqdhb" pod "346fcfc5-c545-4443-aae5-f3945c68b8fe" (UID: "346fcfc5-c545-4443-aae5-f3945c68b8fe"). InnerVolumeSpecName "kube-api-access-cqdhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:01:00.943753 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:00.943727    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/346fcfc5-c545-4443-aae5-f3945c68b8fe-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:01:00.943898 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:00.943821    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqdhb\" (UniqueName: \"kubernetes.io/projected/346fcfc5-c545-4443-aae5-f3945c68b8fe-kube-api-access-cqdhb\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:01:01.548618 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:01.548584    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2fc1b5-mbssl"
Nov 03 09:01:01.548804 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:01.548588    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2fc1b5-mbssl" event={"ID":"346fcfc5-c545-4443-aae5-f3945c68b8fe","Type":"ContainerDied","Data":"83c27cf7a2c58ff7c24b493bcf0d0f5151deb7e011c495f3cca64c3477a4d104"}
Nov 03 09:01:01.548804 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:01.548719    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83c27cf7a2c58ff7c24b493bcf0d0f5151deb7e011c495f3cca64c3477a4d104"
Nov 03 09:01:08.314506 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:08.314471    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3db44a7d-dpkkq"]
Nov 03 09:01:08.315008 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:08.314992    4473 cpu_manager.go:414] "containerMap: removing container" podUID="346fcfc5-c545-4443-aae5-f3945c68b8fe" containerName="revoke-keys"
Nov 03 09:01:08.315066 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:08.315012    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="346fcfc5-c545-4443-aae5-f3945c68b8fe" containerName="revoke-keys"
Nov 03 09:01:08.315117 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:08.315107    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="346fcfc5-c545-4443-aae5-f3945c68b8fe" containerName="revoke-keys"
Nov 03 09:01:08.318359 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:08.318345    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db44a7d-dpkkq"
Nov 03 09:01:08.375890 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:08.375866    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3db44a7d-dpkkq"]
Nov 03 09:01:08.420287 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:08.420259    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p9lq\" (UniqueName: \"kubernetes.io/projected/db1f2d7e-573d-48b5-aa1c-c89f8224797c-kube-api-access-8p9lq\") pod \"maas-api-revoke-sub-3db44a7d-dpkkq\" (UID: \"db1f2d7e-573d-48b5-aa1c-c89f8224797c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db44a7d-dpkkq"
Nov 03 09:01:08.420435 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:08.420292    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db1f2d7e-573d-48b5-aa1c-c89f8224797c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3db44a7d-dpkkq\" (UID: \"db1f2d7e-573d-48b5-aa1c-c89f8224797c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db44a7d-dpkkq"
Nov 03 09:01:08.420435 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:08.420337    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db1f2d7e-573d-48b5-aa1c-c89f8224797c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3db44a7d-dpkkq\" (UID: \"db1f2d7e-573d-48b5-aa1c-c89f8224797c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db44a7d-dpkkq"
Nov 03 09:01:08.422718 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:08.422698    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:01:08.422810 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:08.422698    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:01:08.422810 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:08.422722    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:01:08.422810 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:08.422740    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:01:08.521176 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:08.521145    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p9lq\" (UniqueName: \"kubernetes.io/projected/db1f2d7e-573d-48b5-aa1c-c89f8224797c-kube-api-access-8p9lq\") pod \"maas-api-revoke-sub-3db44a7d-dpkkq\" (UID: \"db1f2d7e-573d-48b5-aa1c-c89f8224797c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db44a7d-dpkkq"
Nov 03 09:01:08.521176 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:08.521178    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db1f2d7e-573d-48b5-aa1c-c89f8224797c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3db44a7d-dpkkq\" (UID: \"db1f2d7e-573d-48b5-aa1c-c89f8224797c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db44a7d-dpkkq"
Nov 03 09:01:08.521417 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:08.521215    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db1f2d7e-573d-48b5-aa1c-c89f8224797c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3db44a7d-dpkkq\" (UID: \"db1f2d7e-573d-48b5-aa1c-c89f8224797c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db44a7d-dpkkq"
Nov 03 09:01:08.522031 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:08.521880    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db1f2d7e-573d-48b5-aa1c-c89f8224797c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3db44a7d-dpkkq\" (UID: \"db1f2d7e-573d-48b5-aa1c-c89f8224797c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db44a7d-dpkkq"
Nov 03 09:01:08.523656 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:08.523635    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db1f2d7e-573d-48b5-aa1c-c89f8224797c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3db44a7d-dpkkq\" (UID: \"db1f2d7e-573d-48b5-aa1c-c89f8224797c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db44a7d-dpkkq"
Nov 03 09:01:08.580641 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:08.580589    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p9lq\" (UniqueName: \"kubernetes.io/projected/db1f2d7e-573d-48b5-aa1c-c89f8224797c-kube-api-access-8p9lq\") pod \"maas-api-revoke-sub-3db44a7d-dpkkq\" (UID: \"db1f2d7e-573d-48b5-aa1c-c89f8224797c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db44a7d-dpkkq"
Nov 03 09:01:08.630593 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:08.630572    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db44a7d-dpkkq"
Nov 03 09:01:09.008160 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:09.008134    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3db44a7d-dpkkq"]
Nov 03 09:01:09.009989 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:01:09.008693    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb1f2d7e_573d_48b5_aa1c_c89f8224797c.slice/crio-9624fdc8ffbbec47ba8a845a8761ebf9651b7f6af2ec2d5211bb829f86f86f3f WatchSource:0}: Error finding container 9624fdc8ffbbec47ba8a845a8761ebf9651b7f6af2ec2d5211bb829f86f86f3f: Status 404 returned error can't find the container with id 9624fdc8ffbbec47ba8a845a8761ebf9651b7f6af2ec2d5211bb829f86f86f3f
Nov 03 09:01:09.599162 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:09.599082    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db44a7d-dpkkq" event={"ID":"db1f2d7e-573d-48b5-aa1c-c89f8224797c","Type":"ContainerStarted","Data":"efe1df3cd7675b92123e2e237a178bd897d197ab249ad4c597e216997da5db55"}
Nov 03 09:01:09.599162 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:09.599131    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db44a7d-dpkkq" event={"ID":"db1f2d7e-573d-48b5-aa1c-c89f8224797c","Type":"ContainerStarted","Data":"9624fdc8ffbbec47ba8a845a8761ebf9651b7f6af2ec2d5211bb829f86f86f3f"}
Nov 03 09:01:09.765911 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:09.765851    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db44a7d-dpkkq" podStartSLOduration=1.7658331299999999 podStartE2EDuration="1.76583313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:01:09.716574112 +0000 UTC m=+1293.985637566" watchObservedRunningTime="2026-11-03 09:01:09.76583313 +0000 UTC m=+1294.034896565"
Nov 03 09:01:09.769084 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:09.769057    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-910d50/e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j"]
Nov 03 09:01:10.608115 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:10.608079    4473 generic.go:356] "Generic (PLEG): container finished" podID="db1f2d7e-573d-48b5-aa1c-c89f8224797c" containerID="efe1df3cd7675b92123e2e237a178bd897d197ab249ad4c597e216997da5db55" exitCode=0
Nov 03 09:01:10.608115 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:10.608124    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db44a7d-dpkkq" event={"ID":"db1f2d7e-573d-48b5-aa1c-c89f8224797c","Type":"ContainerDied","Data":"efe1df3cd7675b92123e2e237a178bd897d197ab249ad4c597e216997da5db55"}
Nov 03 09:01:11.765189 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:11.765163    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db44a7d-dpkkq"
Nov 03 09:01:11.854577 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:11.854541    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db1f2d7e-573d-48b5-aa1c-c89f8224797c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db1f2d7e-573d-48b5-aa1c-c89f8224797c-maas-api-openshift-ca\") pod \"db1f2d7e-573d-48b5-aa1c-c89f8224797c\" (UID: \"db1f2d7e-573d-48b5-aa1c-c89f8224797c\") "
Nov 03 09:01:11.854762 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:11.854634    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db1f2d7e-573d-48b5-aa1c-c89f8224797c-kube-api-access-8p9lq\" (UniqueName: \"kubernetes.io/projected/db1f2d7e-573d-48b5-aa1c-c89f8224797c-kube-api-access-8p9lq\") pod \"db1f2d7e-573d-48b5-aa1c-c89f8224797c\" (UID: \"db1f2d7e-573d-48b5-aa1c-c89f8224797c\") "
Nov 03 09:01:11.854833 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:11.854814    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db1f2d7e-573d-48b5-aa1c-c89f8224797c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db1f2d7e-573d-48b5-aa1c-c89f8224797c-maas-api-xks-ca\") pod \"db1f2d7e-573d-48b5-aa1c-c89f8224797c\" (UID: \"db1f2d7e-573d-48b5-aa1c-c89f8224797c\") "
Nov 03 09:01:11.854889 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:11.854858    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db1f2d7e-573d-48b5-aa1c-c89f8224797c-maas-api-openshift-ca" pod "db1f2d7e-573d-48b5-aa1c-c89f8224797c" (UID: "db1f2d7e-573d-48b5-aa1c-c89f8224797c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:01:11.855263 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:11.855245    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db1f2d7e-573d-48b5-aa1c-c89f8224797c-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:01:11.856976 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:11.856954    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db1f2d7e-573d-48b5-aa1c-c89f8224797c-maas-api-xks-ca" pod "db1f2d7e-573d-48b5-aa1c-c89f8224797c" (UID: "db1f2d7e-573d-48b5-aa1c-c89f8224797c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:01:11.856976 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:11.856957    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db1f2d7e-573d-48b5-aa1c-c89f8224797c-kube-api-access-8p9lq" pod "db1f2d7e-573d-48b5-aa1c-c89f8224797c" (UID: "db1f2d7e-573d-48b5-aa1c-c89f8224797c"). InnerVolumeSpecName "kube-api-access-8p9lq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:01:11.956411 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:11.956376    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p9lq\" (UniqueName: \"kubernetes.io/projected/db1f2d7e-573d-48b5-aa1c-c89f8224797c-kube-api-access-8p9lq\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:01:11.956411 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:11.956411    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db1f2d7e-573d-48b5-aa1c-c89f8224797c-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:01:12.054814 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:12.054774    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7639d7e8-zwlsr"]
Nov 03 09:01:12.057443 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:12.057413    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7639d7e8-zwlsr"]
Nov 03 09:01:12.624938 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:12.624903    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db44a7d-dpkkq" event={"ID":"db1f2d7e-573d-48b5-aa1c-c89f8224797c","Type":"ContainerDied","Data":"9624fdc8ffbbec47ba8a845a8761ebf9651b7f6af2ec2d5211bb829f86f86f3f"}
Nov 03 09:01:12.624938 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:12.624936    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db44a7d-dpkkq"
Nov 03 09:01:12.625160 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:12.625012    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9624fdc8ffbbec47ba8a845a8761ebf9651b7f6af2ec2d5211bb829f86f86f3f"
Nov 03 09:01:13.590107 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:13.590074    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx"]
Nov 03 09:01:13.590578 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:13.590564    4473 cpu_manager.go:414] "containerMap: removing container" podUID="db1f2d7e-573d-48b5-aa1c-c89f8224797c" containerName="revoke-keys"
Nov 03 09:01:13.590637 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:13.590580    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db1f2d7e-573d-48b5-aa1c-c89f8224797c" containerName="revoke-keys"
Nov 03 09:01:13.590669 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:13.590652    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="db1f2d7e-573d-48b5-aa1c-c89f8224797c" containerName="revoke-keys"
Nov 03 09:01:13.593814 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:13.593796    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx"
Nov 03 09:01:13.675595 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:13.675554    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e84a474-def6-41dc-8516-5e5ccf42ff46-model-cache\") pod \"distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx\" (UID: \"8e84a474-def6-41dc-8516-5e5ccf42ff46\") " pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx"
Nov 03 09:01:13.675758 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:13.675621    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e84a474-def6-41dc-8516-5e5ccf42ff46-tls-certs\") pod \"distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx\" (UID: \"8e84a474-def6-41dc-8516-5e5ccf42ff46\") " pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx"
Nov 03 09:01:13.675758 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:13.675695    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e84a474-def6-41dc-8516-5e5ccf42ff46-home\") pod \"distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx\" (UID: \"8e84a474-def6-41dc-8516-5e5ccf42ff46\") " pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx"
Nov 03 09:01:13.675758 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:13.675741    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26jz6\" (UniqueName: \"kubernetes.io/projected/8e84a474-def6-41dc-8516-5e5ccf42ff46-kube-api-access-26jz6\") pod \"distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx\" (UID: \"8e84a474-def6-41dc-8516-5e5ccf42ff46\") " pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx"
Nov 03 09:01:13.675857 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:13.675838    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e84a474-def6-41dc-8516-5e5ccf42ff46-tmp-dir\") pod \"distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx\" (UID: \"8e84a474-def6-41dc-8516-5e5ccf42ff46\") " pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx"
Nov 03 09:01:13.675909 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:13.675896    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e84a474-def6-41dc-8516-5e5ccf42ff46-dshm\") pod \"distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx\" (UID: \"8e84a474-def6-41dc-8516-5e5ccf42ff46\") " pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx"
Nov 03 09:01:13.698042 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:13.698019    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx"]
Nov 03 09:01:13.699025 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:13.699009    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-98d5dd\"/\"distinct-2-w3-98d5dd-kserve-self-signed-certs\""
Nov 03 09:01:13.777929 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:13.776807    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e84a474-def6-41dc-8516-5e5ccf42ff46-dshm\") pod \"distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx\" (UID: \"8e84a474-def6-41dc-8516-5e5ccf42ff46\") " pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx"
Nov 03 09:01:13.777929 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:13.776919    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e84a474-def6-41dc-8516-5e5ccf42ff46-model-cache\") pod \"distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx\" (UID: \"8e84a474-def6-41dc-8516-5e5ccf42ff46\") " pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx"
Nov 03 09:01:13.777929 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:13.777029    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e84a474-def6-41dc-8516-5e5ccf42ff46-tls-certs\") pod \"distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx\" (UID: \"8e84a474-def6-41dc-8516-5e5ccf42ff46\") " pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx"
Nov 03 09:01:13.777929 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:13.777060    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e84a474-def6-41dc-8516-5e5ccf42ff46-home\") pod \"distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx\" (UID: \"8e84a474-def6-41dc-8516-5e5ccf42ff46\") " pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx"
Nov 03 09:01:13.777929 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:13.777109    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26jz6\" (UniqueName: \"kubernetes.io/projected/8e84a474-def6-41dc-8516-5e5ccf42ff46-kube-api-access-26jz6\") pod \"distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx\" (UID: \"8e84a474-def6-41dc-8516-5e5ccf42ff46\") " pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx"
Nov 03 09:01:13.777929 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:13.777151    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e84a474-def6-41dc-8516-5e5ccf42ff46-tmp-dir\") pod \"distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx\" (UID: \"8e84a474-def6-41dc-8516-5e5ccf42ff46\") " pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx"
Nov 03 09:01:13.777929 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:13.777615    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e84a474-def6-41dc-8516-5e5ccf42ff46-tmp-dir\") pod \"distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx\" (UID: \"8e84a474-def6-41dc-8516-5e5ccf42ff46\") " pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx"
Nov 03 09:01:13.795892 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:13.795856    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e84a474-def6-41dc-8516-5e5ccf42ff46-home\") pod \"distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx\" (UID: \"8e84a474-def6-41dc-8516-5e5ccf42ff46\") " pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx"
Nov 03 09:01:13.796386 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:13.796355    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e84a474-def6-41dc-8516-5e5ccf42ff46-model-cache\") pod \"distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx\" (UID: \"8e84a474-def6-41dc-8516-5e5ccf42ff46\") " pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx"
Nov 03 09:01:13.797081 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:13.797034    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e84a474-def6-41dc-8516-5e5ccf42ff46-dshm\") pod \"distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx\" (UID: \"8e84a474-def6-41dc-8516-5e5ccf42ff46\") " pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx"
Nov 03 09:01:13.799601 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:13.799577    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e84a474-def6-41dc-8516-5e5ccf42ff46-tls-certs\") pod \"distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx\" (UID: \"8e84a474-def6-41dc-8516-5e5ccf42ff46\") " pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx"
Nov 03 09:01:13.857154 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:13.857090    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26jz6\" (UniqueName: \"kubernetes.io/projected/8e84a474-def6-41dc-8516-5e5ccf42ff46-kube-api-access-26jz6\") pod \"distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx\" (UID: \"8e84a474-def6-41dc-8516-5e5ccf42ff46\") " pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx"
Nov 03 09:01:13.870694 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:13.869971    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07d5d4c6-25ee-4574-b508-7c9ef5b3be45" path="/var/lib/kubelet/pods/07d5d4c6-25ee-4574-b508-7c9ef5b3be45/volumes"
Nov 03 09:01:13.905187 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:13.905144    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx"
Nov 03 09:01:14.380808 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:14.380783    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx"]
Nov 03 09:01:14.381610 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:01:14.381586    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e84a474_def6_41dc_8516_5e5ccf42ff46.slice/crio-03d355bc945204293c3919215c0aa6e198d2d028e498b1b9b4a7732f8fdcfa6b WatchSource:0}: Error finding container 03d355bc945204293c3919215c0aa6e198d2d028e498b1b9b4a7732f8fdcfa6b: Status 404 returned error can't find the container with id 03d355bc945204293c3919215c0aa6e198d2d028e498b1b9b4a7732f8fdcfa6b
Nov 03 09:01:14.638669 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:14.638589    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx" event={"ID":"8e84a474-def6-41dc-8516-5e5ccf42ff46","Type":"ContainerStarted","Data":"def8668025fa79aaae2cca1fd592da75f74871183aad2ab4bbbf33264f79a5c0"}
Nov 03 09:01:14.638669 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:14.638632    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx" event={"ID":"8e84a474-def6-41dc-8516-5e5ccf42ff46","Type":"ContainerStarted","Data":"03d355bc945204293c3919215c0aa6e198d2d028e498b1b9b4a7732f8fdcfa6b"}
Nov 03 09:01:14.757552 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:14.757501    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx" podStartSLOduration=1.7574865210000001 podStartE2EDuration="1.757486521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:01:14.756321759 +0000 UTC m=+1299.025385210" watchObservedRunningTime="2026-11-03 09:01:14.757486521 +0000 UTC m=+1299.026549954"
Nov 03 09:01:14.913044 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:14.909635    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx"
Nov 03 09:01:15.157542 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:15.157515    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cbe5945-l5pxb"]
Nov 03 09:01:15.161257 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:15.161242    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbe5945-l5pxb"
Nov 03 09:01:15.215923 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:15.215882    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:01:15.218926 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:15.218898    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cbe5945-l5pxb"]
Nov 03 09:01:15.265262 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:15.265231    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:01:15.265262 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:15.265248    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:01:15.265478 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:15.265345    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:01:15.297004 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:15.296976    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cbe5945-l5pxb\" (UID: \"bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbe5945-l5pxb"
Nov 03 09:01:15.297157 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:15.297024    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cbe5945-l5pxb\" (UID: \"bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbe5945-l5pxb"
Nov 03 09:01:15.297228 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:15.297171    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzbm5\" (UniqueName: \"kubernetes.io/projected/bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d-kube-api-access-hzbm5\") pod \"maas-api-revoke-sub-4cbe5945-l5pxb\" (UID: \"bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbe5945-l5pxb"
Nov 03 09:01:15.398162 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:15.398133    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cbe5945-l5pxb\" (UID: \"bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbe5945-l5pxb"
Nov 03 09:01:15.398334 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:15.398186    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cbe5945-l5pxb\" (UID: \"bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbe5945-l5pxb"
Nov 03 09:01:15.398334 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:15.398243    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzbm5\" (UniqueName: \"kubernetes.io/projected/bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d-kube-api-access-hzbm5\") pod \"maas-api-revoke-sub-4cbe5945-l5pxb\" (UID: \"bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbe5945-l5pxb"
Nov 03 09:01:15.398878 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:15.398854    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cbe5945-l5pxb\" (UID: \"bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbe5945-l5pxb"
Nov 03 09:01:15.400720 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:15.400682    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cbe5945-l5pxb\" (UID: \"bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbe5945-l5pxb"
Nov 03 09:01:15.458372 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:15.458309    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzbm5\" (UniqueName: \"kubernetes.io/projected/bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d-kube-api-access-hzbm5\") pod \"maas-api-revoke-sub-4cbe5945-l5pxb\" (UID: \"bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbe5945-l5pxb"
Nov 03 09:01:15.474745 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:15.474718    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbe5945-l5pxb"
Nov 03 09:01:15.851215 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:15.851182    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cbe5945-l5pxb"]
Nov 03 09:01:15.851924 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:01:15.851898    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfd4a6a8_61ed_44ca_acaf_6cbe6455bb4d.slice/crio-1772be4c28dec16493db3b3c0c2ce07aaf632c097e923bf2232f9bb21e3d4201 WatchSource:0}: Error finding container 1772be4c28dec16493db3b3c0c2ce07aaf632c097e923bf2232f9bb21e3d4201: Status 404 returned error can't find the container with id 1772be4c28dec16493db3b3c0c2ce07aaf632c097e923bf2232f9bb21e3d4201
Nov 03 09:01:16.652004 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:16.651962    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbe5945-l5pxb" event={"ID":"bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d","Type":"ContainerStarted","Data":"f5925db1ccbd921e3ae4021c47301bf3a1de9a7fabb923a6d63449574c43e9ca"}
Nov 03 09:01:16.652220 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:16.652032    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbe5945-l5pxb" event={"ID":"bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d","Type":"ContainerStarted","Data":"1772be4c28dec16493db3b3c0c2ce07aaf632c097e923bf2232f9bb21e3d4201"}
Nov 03 09:01:16.769416 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:16.769360    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbe5945-l5pxb" podStartSLOduration=1.769348291 podStartE2EDuration="1.769348291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:01:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:01:16.76705581 +0000 UTC m=+1301.036119298" watchObservedRunningTime="2026-11-03 09:01:16.769348291 +0000 UTC m=+1301.038411724"
Nov 03 09:01:16.937614 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:01:16.937508    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:01:17.479719 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:17.479686    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0be8722-6lsc2"]
Nov 03 09:01:17.483303 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:17.483288    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0be8722-6lsc2"
Nov 03 09:01:17.541895 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:17.541861    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0be8722-6lsc2"]
Nov 03 09:01:17.628838 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:17.628802    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z6pk\" (UniqueName: \"kubernetes.io/projected/7e8747ce-e051-435a-a5d9-bdd4e3991391-kube-api-access-5z6pk\") pod \"maas-api-revoke-sub-d0be8722-6lsc2\" (UID: \"7e8747ce-e051-435a-a5d9-bdd4e3991391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0be8722-6lsc2"
Nov 03 09:01:17.629029 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:17.628979    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e8747ce-e051-435a-a5d9-bdd4e3991391-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0be8722-6lsc2\" (UID: \"7e8747ce-e051-435a-a5d9-bdd4e3991391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0be8722-6lsc2"
Nov 03 09:01:17.629082 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:17.629042    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e8747ce-e051-435a-a5d9-bdd4e3991391-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0be8722-6lsc2\" (UID: \"7e8747ce-e051-435a-a5d9-bdd4e3991391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0be8722-6lsc2"
Nov 03 09:01:17.658801 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:17.658770    4473 generic.go:356] "Generic (PLEG): container finished" podID="bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d" containerID="f5925db1ccbd921e3ae4021c47301bf3a1de9a7fabb923a6d63449574c43e9ca" exitCode=0
Nov 03 09:01:17.658982 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:17.658859    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbe5945-l5pxb" event={"ID":"bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d","Type":"ContainerDied","Data":"f5925db1ccbd921e3ae4021c47301bf3a1de9a7fabb923a6d63449574c43e9ca"}
Nov 03 09:01:17.729971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:17.729872    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e8747ce-e051-435a-a5d9-bdd4e3991391-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0be8722-6lsc2\" (UID: \"7e8747ce-e051-435a-a5d9-bdd4e3991391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0be8722-6lsc2"
Nov 03 09:01:17.729971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:17.729923    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e8747ce-e051-435a-a5d9-bdd4e3991391-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0be8722-6lsc2\" (UID: \"7e8747ce-e051-435a-a5d9-bdd4e3991391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0be8722-6lsc2"
Nov 03 09:01:17.729971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:17.729968    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z6pk\" (UniqueName: \"kubernetes.io/projected/7e8747ce-e051-435a-a5d9-bdd4e3991391-kube-api-access-5z6pk\") pod \"maas-api-revoke-sub-d0be8722-6lsc2\" (UID: \"7e8747ce-e051-435a-a5d9-bdd4e3991391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0be8722-6lsc2"
Nov 03 09:01:17.730590 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:17.730563    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e8747ce-e051-435a-a5d9-bdd4e3991391-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0be8722-6lsc2\" (UID: \"7e8747ce-e051-435a-a5d9-bdd4e3991391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0be8722-6lsc2"
Nov 03 09:01:17.732494 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:17.732475    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e8747ce-e051-435a-a5d9-bdd4e3991391-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0be8722-6lsc2\" (UID: \"7e8747ce-e051-435a-a5d9-bdd4e3991391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0be8722-6lsc2"
Nov 03 09:01:17.789220 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:17.789194    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z6pk\" (UniqueName: \"kubernetes.io/projected/7e8747ce-e051-435a-a5d9-bdd4e3991391-kube-api-access-5z6pk\") pod \"maas-api-revoke-sub-d0be8722-6lsc2\" (UID: \"7e8747ce-e051-435a-a5d9-bdd4e3991391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0be8722-6lsc2"
Nov 03 09:01:17.796094 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:17.796074    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0be8722-6lsc2"
Nov 03 09:01:18.053427 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:18.053341    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6427e717-nvch9"]
Nov 03 09:01:18.057100 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:18.057073    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6427e717-nvch9"]
Nov 03 09:01:18.181039 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:18.181011    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0be8722-6lsc2"]
Nov 03 09:01:18.181628 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:01:18.181596    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e8747ce_e051_435a_a5d9_bdd4e3991391.slice/crio-a4afec5f7820bea1136afe29186edd0051c8e90e3637447d08366cb295f7bfff WatchSource:0}: Error finding container a4afec5f7820bea1136afe29186edd0051c8e90e3637447d08366cb295f7bfff: Status 404 returned error can't find the container with id a4afec5f7820bea1136afe29186edd0051c8e90e3637447d08366cb295f7bfff
Nov 03 09:01:18.666618 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:18.666584    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0be8722-6lsc2" event={"ID":"7e8747ce-e051-435a-a5d9-bdd4e3991391","Type":"ContainerStarted","Data":"d1c7755d0d14f058f75b73470d5d238917b4e1dc76ecf55bd82795f54a4a141a"}
Nov 03 09:01:18.666800 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:18.666645    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0be8722-6lsc2" event={"ID":"7e8747ce-e051-435a-a5d9-bdd4e3991391","Type":"ContainerStarted","Data":"a4afec5f7820bea1136afe29186edd0051c8e90e3637447d08366cb295f7bfff"}
Nov 03 09:01:18.783786 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:18.783740    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0be8722-6lsc2" podStartSLOduration=1.783725746 podStartE2EDuration="1.783725746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:01:18.782472288 +0000 UTC m=+1303.051535746" watchObservedRunningTime="2026-11-03 09:01:18.783725746 +0000 UTC m=+1303.052789221"
Nov 03 09:01:18.803496 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:18.803473    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbe5945-l5pxb"
Nov 03 09:01:18.942377 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:18.942296    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d-kube-api-access-hzbm5\" (UniqueName: \"kubernetes.io/projected/bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d-kube-api-access-hzbm5\") pod \"bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d\" (UID: \"bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d\") "
Nov 03 09:01:18.942533 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:18.942388    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d-maas-api-openshift-ca\") pod \"bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d\" (UID: \"bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d\") "
Nov 03 09:01:18.942533 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:18.942463    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d-maas-api-xks-ca\") pod \"bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d\" (UID: \"bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d\") "
Nov 03 09:01:18.942739 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:18.942714    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d-maas-api-openshift-ca" pod "bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d" (UID: "bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:01:18.942889 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:18.942872    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:01:18.944696 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:18.944665    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d-kube-api-access-hzbm5" pod "bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d" (UID: "bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d"). InnerVolumeSpecName "kube-api-access-hzbm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:01:18.944818 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:18.944695    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d-maas-api-xks-ca" pod "bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d" (UID: "bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:01:19.043993 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:19.043933    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:01:19.044156 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:19.043996    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzbm5\" (UniqueName: \"kubernetes.io/projected/bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d-kube-api-access-hzbm5\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:01:19.674416 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:19.674381    4473 generic.go:356] "Generic (PLEG): container finished" podID="7e8747ce-e051-435a-a5d9-bdd4e3991391" containerID="d1c7755d0d14f058f75b73470d5d238917b4e1dc76ecf55bd82795f54a4a141a" exitCode=0
Nov 03 09:01:19.674835 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:19.674461    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0be8722-6lsc2" event={"ID":"7e8747ce-e051-435a-a5d9-bdd4e3991391","Type":"ContainerDied","Data":"d1c7755d0d14f058f75b73470d5d238917b4e1dc76ecf55bd82795f54a4a141a"}
Nov 03 09:01:19.675843 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:19.675821    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbe5945-l5pxb" event={"ID":"bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d","Type":"ContainerDied","Data":"1772be4c28dec16493db3b3c0c2ce07aaf632c097e923bf2232f9bb21e3d4201"}
Nov 03 09:01:19.675843 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:19.675840    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cbe5945-l5pxb"
Nov 03 09:01:19.676052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:19.675856    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1772be4c28dec16493db3b3c0c2ce07aaf632c097e923bf2232f9bb21e3d4201"
Nov 03 09:01:19.873601 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:19.873566    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aa5fad9-a1bc-4e3a-8d09-226887ac86c5" path="/var/lib/kubelet/pods/1aa5fad9-a1bc-4e3a-8d09-226887ac86c5/volumes"
Nov 03 09:01:20.822727 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:20.822703    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0be8722-6lsc2"
Nov 03 09:01:20.965487 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:20.965412    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e8747ce-e051-435a-a5d9-bdd4e3991391-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e8747ce-e051-435a-a5d9-bdd4e3991391-maas-api-xks-ca\") pod \"7e8747ce-e051-435a-a5d9-bdd4e3991391\" (UID: \"7e8747ce-e051-435a-a5d9-bdd4e3991391\") "
Nov 03 09:01:20.965625 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:20.965483    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e8747ce-e051-435a-a5d9-bdd4e3991391-kube-api-access-5z6pk\" (UniqueName: \"kubernetes.io/projected/7e8747ce-e051-435a-a5d9-bdd4e3991391-kube-api-access-5z6pk\") pod \"7e8747ce-e051-435a-a5d9-bdd4e3991391\" (UID: \"7e8747ce-e051-435a-a5d9-bdd4e3991391\") "
Nov 03 09:01:20.965625 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:20.965556    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e8747ce-e051-435a-a5d9-bdd4e3991391-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e8747ce-e051-435a-a5d9-bdd4e3991391-maas-api-openshift-ca\") pod \"7e8747ce-e051-435a-a5d9-bdd4e3991391\" (UID: \"7e8747ce-e051-435a-a5d9-bdd4e3991391\") "
Nov 03 09:01:20.966021 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:20.965990    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e8747ce-e051-435a-a5d9-bdd4e3991391-maas-api-openshift-ca" pod "7e8747ce-e051-435a-a5d9-bdd4e3991391" (UID: "7e8747ce-e051-435a-a5d9-bdd4e3991391"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:01:20.966255 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:20.966234    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e8747ce-e051-435a-a5d9-bdd4e3991391-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:01:20.967732 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:20.967709    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e8747ce-e051-435a-a5d9-bdd4e3991391-maas-api-xks-ca" pod "7e8747ce-e051-435a-a5d9-bdd4e3991391" (UID: "7e8747ce-e051-435a-a5d9-bdd4e3991391"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:01:20.967831 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:20.967777    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e8747ce-e051-435a-a5d9-bdd4e3991391-kube-api-access-5z6pk" pod "7e8747ce-e051-435a-a5d9-bdd4e3991391" (UID: "7e8747ce-e051-435a-a5d9-bdd4e3991391"). InnerVolumeSpecName "kube-api-access-5z6pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:01:21.067049 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:21.067020    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e8747ce-e051-435a-a5d9-bdd4e3991391-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:01:21.067049 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:21.067044    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z6pk\" (UniqueName: \"kubernetes.io/projected/7e8747ce-e051-435a-a5d9-bdd4e3991391-kube-api-access-5z6pk\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:01:21.100013 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:21.099984    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:01:21.721968 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:21.718176    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0be8722-6lsc2" event={"ID":"7e8747ce-e051-435a-a5d9-bdd4e3991391","Type":"ContainerDied","Data":"a4afec5f7820bea1136afe29186edd0051c8e90e3637447d08366cb295f7bfff"}
Nov 03 09:01:21.721968 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:21.718264    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4afec5f7820bea1136afe29186edd0051c8e90e3637447d08366cb295f7bfff"
Nov 03 09:01:21.721968 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:21.718344    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0be8722-6lsc2"
Nov 03 09:01:23.906348 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:23.906311    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx"
Nov 03 09:01:23.919120 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:23.919100    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx"
Nov 03 09:01:24.398226 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:24.398189    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn"]
Nov 03 09:01:24.398683 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:24.398670    4473 cpu_manager.go:414] "containerMap: removing container" podUID="7e8747ce-e051-435a-a5d9-bdd4e3991391" containerName="revoke-keys"
Nov 03 09:01:24.398728 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:24.398685    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e8747ce-e051-435a-a5d9-bdd4e3991391" containerName="revoke-keys"
Nov 03 09:01:24.398728 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:24.398714    4473 cpu_manager.go:414] "containerMap: removing container" podUID="bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d" containerName="revoke-keys"
Nov 03 09:01:24.398728 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:24.398719    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d" containerName="revoke-keys"
Nov 03 09:01:24.398822 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:24.398788    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d" containerName="revoke-keys"
Nov 03 09:01:24.398822 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:24.398800    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e8747ce-e051-435a-a5d9-bdd4e3991391" containerName="revoke-keys"
Nov 03 09:01:24.402237 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:24.402222    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn"
Nov 03 09:01:24.425382 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:24.425353    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/035b0461-2dfa-42f4-8ea7-2d1911cddca9-home\") pod \"distinct-w0-910d50-kserve-679d4f866-jxdtn\" (UID: \"035b0461-2dfa-42f4-8ea7-2d1911cddca9\") " pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn"
Nov 03 09:01:24.425532 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:24.425408    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/035b0461-2dfa-42f4-8ea7-2d1911cddca9-dshm\") pod \"distinct-w0-910d50-kserve-679d4f866-jxdtn\" (UID: \"035b0461-2dfa-42f4-8ea7-2d1911cddca9\") " pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn"
Nov 03 09:01:24.425532 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:24.425427    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/035b0461-2dfa-42f4-8ea7-2d1911cddca9-tmp-dir\") pod \"distinct-w0-910d50-kserve-679d4f866-jxdtn\" (UID: \"035b0461-2dfa-42f4-8ea7-2d1911cddca9\") " pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn"
Nov 03 09:01:24.425532 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:24.425478    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/035b0461-2dfa-42f4-8ea7-2d1911cddca9-tls-certs\") pod \"distinct-w0-910d50-kserve-679d4f866-jxdtn\" (UID: \"035b0461-2dfa-42f4-8ea7-2d1911cddca9\") " pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn"
Nov 03 09:01:24.425660 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:24.425531    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mcnc\" (UniqueName: \"kubernetes.io/projected/035b0461-2dfa-42f4-8ea7-2d1911cddca9-kube-api-access-8mcnc\") pod \"distinct-w0-910d50-kserve-679d4f866-jxdtn\" (UID: \"035b0461-2dfa-42f4-8ea7-2d1911cddca9\") " pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn"
Nov 03 09:01:24.425696 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:24.425668    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/035b0461-2dfa-42f4-8ea7-2d1911cddca9-model-cache\") pod \"distinct-w0-910d50-kserve-679d4f866-jxdtn\" (UID: \"035b0461-2dfa-42f4-8ea7-2d1911cddca9\") " pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn"
Nov 03 09:01:24.462279 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:24.462254    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-910d50\"/\"distinct-w0-910d50-kserve-self-signed-certs\""
Nov 03 09:01:24.512590 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:24.512551    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn"]
Nov 03 09:01:24.526525 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:24.526491    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/035b0461-2dfa-42f4-8ea7-2d1911cddca9-dshm\") pod \"distinct-w0-910d50-kserve-679d4f866-jxdtn\" (UID: \"035b0461-2dfa-42f4-8ea7-2d1911cddca9\") " pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn"
Nov 03 09:01:24.526664 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:24.526534    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/035b0461-2dfa-42f4-8ea7-2d1911cddca9-tmp-dir\") pod \"distinct-w0-910d50-kserve-679d4f866-jxdtn\" (UID: \"035b0461-2dfa-42f4-8ea7-2d1911cddca9\") " pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn"
Nov 03 09:01:24.526664 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:24.526558    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/035b0461-2dfa-42f4-8ea7-2d1911cddca9-tls-certs\") pod \"distinct-w0-910d50-kserve-679d4f866-jxdtn\" (UID: \"035b0461-2dfa-42f4-8ea7-2d1911cddca9\") " pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn"
Nov 03 09:01:24.526664 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:24.526587    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mcnc\" (UniqueName: \"kubernetes.io/projected/035b0461-2dfa-42f4-8ea7-2d1911cddca9-kube-api-access-8mcnc\") pod \"distinct-w0-910d50-kserve-679d4f866-jxdtn\" (UID: \"035b0461-2dfa-42f4-8ea7-2d1911cddca9\") " pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn"
Nov 03 09:01:24.526664 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:24.526649    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/035b0461-2dfa-42f4-8ea7-2d1911cddca9-model-cache\") pod \"distinct-w0-910d50-kserve-679d4f866-jxdtn\" (UID: \"035b0461-2dfa-42f4-8ea7-2d1911cddca9\") " pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn"
Nov 03 09:01:24.526869 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:24.526796    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/035b0461-2dfa-42f4-8ea7-2d1911cddca9-home\") pod \"distinct-w0-910d50-kserve-679d4f866-jxdtn\" (UID: \"035b0461-2dfa-42f4-8ea7-2d1911cddca9\") " pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn"
Nov 03 09:01:24.526994 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:24.526971    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/035b0461-2dfa-42f4-8ea7-2d1911cddca9-tmp-dir\") pod \"distinct-w0-910d50-kserve-679d4f866-jxdtn\" (UID: \"035b0461-2dfa-42f4-8ea7-2d1911cddca9\") " pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn"
Nov 03 09:01:24.527122 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:24.527102    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/035b0461-2dfa-42f4-8ea7-2d1911cddca9-model-cache\") pod \"distinct-w0-910d50-kserve-679d4f866-jxdtn\" (UID: \"035b0461-2dfa-42f4-8ea7-2d1911cddca9\") " pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn"
Nov 03 09:01:24.527201 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:24.527172    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/035b0461-2dfa-42f4-8ea7-2d1911cddca9-home\") pod \"distinct-w0-910d50-kserve-679d4f866-jxdtn\" (UID: \"035b0461-2dfa-42f4-8ea7-2d1911cddca9\") " pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn"
Nov 03 09:01:24.529124 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:24.529100    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/035b0461-2dfa-42f4-8ea7-2d1911cddca9-dshm\") pod \"distinct-w0-910d50-kserve-679d4f866-jxdtn\" (UID: \"035b0461-2dfa-42f4-8ea7-2d1911cddca9\") " pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn"
Nov 03 09:01:24.529280 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:24.529262    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/035b0461-2dfa-42f4-8ea7-2d1911cddca9-tls-certs\") pod \"distinct-w0-910d50-kserve-679d4f866-jxdtn\" (UID: \"035b0461-2dfa-42f4-8ea7-2d1911cddca9\") " pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn"
Nov 03 09:01:24.594901 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:24.594871    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mcnc\" (UniqueName: \"kubernetes.io/projected/035b0461-2dfa-42f4-8ea7-2d1911cddca9-kube-api-access-8mcnc\") pod \"distinct-w0-910d50-kserve-679d4f866-jxdtn\" (UID: \"035b0461-2dfa-42f4-8ea7-2d1911cddca9\") " pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn"
Nov 03 09:01:24.714229 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:24.714137    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn"
Nov 03 09:01:24.757862 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:24.757837    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx"
Nov 03 09:01:24.917365 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:24.917299    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j" podUID="0f1584e1-c962-4d28-8c23-7885eccc45f6" containerName="main" containerID="cri-o://4e2536c542b85ca9b67e7f27b5d47010600c2ede499eb0447ae78ca875ffae31" gracePeriod=45
Nov 03 09:01:24.979627 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:24.979558    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-224496da-f7xt7"]
Nov 03 09:01:24.983191 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:24.983173    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-224496da-f7xt7"
Nov 03 09:01:25.008188 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.008157    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j" podUID="0f1584e1-c962-4d28-8c23-7885eccc45f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": dial tcp 10.133.0.55:8000: connect: connection refused"
Nov 03 09:01:25.037835 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.037810    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:01:25.038014 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.037880    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:01:25.038133 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.038071    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:01:25.038311 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.038290    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:01:25.083120 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.083090    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-224496da-f7xt7"]
Nov 03 09:01:25.122853 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.122502    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e697407f-b6fb-45d7-bc7f-743879e389c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-224496da-f7xt7\" (UID: \"e697407f-b6fb-45d7-bc7f-743879e389c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-224496da-f7xt7"
Nov 03 09:01:25.122853 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.122553    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e697407f-b6fb-45d7-bc7f-743879e389c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-224496da-f7xt7\" (UID: \"e697407f-b6fb-45d7-bc7f-743879e389c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-224496da-f7xt7"
Nov 03 09:01:25.122853 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.122693    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpnq9\" (UniqueName: \"kubernetes.io/projected/e697407f-b6fb-45d7-bc7f-743879e389c5-kube-api-access-fpnq9\") pod \"maas-api-revoke-sub-224496da-f7xt7\" (UID: \"e697407f-b6fb-45d7-bc7f-743879e389c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-224496da-f7xt7"
Nov 03 09:01:25.129656 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.129636    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn"]
Nov 03 09:01:25.200549 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.200527    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j"
Nov 03 09:01:25.223814 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.223789    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f1584e1-c962-4d28-8c23-7885eccc45f6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f1584e1-c962-4d28-8c23-7885eccc45f6-dshm\") pod \"0f1584e1-c962-4d28-8c23-7885eccc45f6\" (UID: \"0f1584e1-c962-4d28-8c23-7885eccc45f6\") "
Nov 03 09:01:25.223990 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.223914    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f1584e1-c962-4d28-8c23-7885eccc45f6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f1584e1-c962-4d28-8c23-7885eccc45f6-model-cache\") pod \"0f1584e1-c962-4d28-8c23-7885eccc45f6\" (UID: \"0f1584e1-c962-4d28-8c23-7885eccc45f6\") "
Nov 03 09:01:25.223990 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.223940    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f1584e1-c962-4d28-8c23-7885eccc45f6-kube-api-access-k9d7v\" (UniqueName: \"kubernetes.io/projected/0f1584e1-c962-4d28-8c23-7885eccc45f6-kube-api-access-k9d7v\") pod \"0f1584e1-c962-4d28-8c23-7885eccc45f6\" (UID: \"0f1584e1-c962-4d28-8c23-7885eccc45f6\") "
Nov 03 09:01:25.224127 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.224020    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f1584e1-c962-4d28-8c23-7885eccc45f6-tls-certs\" (UniqueName: \"kubernetes.io/secret/0f1584e1-c962-4d28-8c23-7885eccc45f6-tls-certs\") pod \"0f1584e1-c962-4d28-8c23-7885eccc45f6\" (UID: \"0f1584e1-c962-4d28-8c23-7885eccc45f6\") "
Nov 03 09:01:25.224127 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.224093    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f1584e1-c962-4d28-8c23-7885eccc45f6-home\" (UniqueName: \"kubernetes.io/empty-dir/0f1584e1-c962-4d28-8c23-7885eccc45f6-home\") pod \"0f1584e1-c962-4d28-8c23-7885eccc45f6\" (UID: \"0f1584e1-c962-4d28-8c23-7885eccc45f6\") "
Nov 03 09:01:25.224127 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.224113    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f1584e1-c962-4d28-8c23-7885eccc45f6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f1584e1-c962-4d28-8c23-7885eccc45f6-tmp-dir\") pod \"0f1584e1-c962-4d28-8c23-7885eccc45f6\" (UID: \"0f1584e1-c962-4d28-8c23-7885eccc45f6\") "
Nov 03 09:01:25.224345 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.224322    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpnq9\" (UniqueName: \"kubernetes.io/projected/e697407f-b6fb-45d7-bc7f-743879e389c5-kube-api-access-fpnq9\") pod \"maas-api-revoke-sub-224496da-f7xt7\" (UID: \"e697407f-b6fb-45d7-bc7f-743879e389c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-224496da-f7xt7"
Nov 03 09:01:25.224466 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.224436    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e697407f-b6fb-45d7-bc7f-743879e389c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-224496da-f7xt7\" (UID: \"e697407f-b6fb-45d7-bc7f-743879e389c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-224496da-f7xt7"
Nov 03 09:01:25.224525 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.224488    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e697407f-b6fb-45d7-bc7f-743879e389c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-224496da-f7xt7\" (UID: \"e697407f-b6fb-45d7-bc7f-743879e389c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-224496da-f7xt7"
Nov 03 09:01:25.224525 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.224501    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f1584e1-c962-4d28-8c23-7885eccc45f6-home" pod "0f1584e1-c962-4d28-8c23-7885eccc45f6" (UID: "0f1584e1-c962-4d28-8c23-7885eccc45f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:01:25.224740 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.224721    4473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f1584e1-c962-4d28-8c23-7885eccc45f6-home\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:01:25.224912 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.224856    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f1584e1-c962-4d28-8c23-7885eccc45f6-tmp-dir" pod "0f1584e1-c962-4d28-8c23-7885eccc45f6" (UID: "0f1584e1-c962-4d28-8c23-7885eccc45f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:01:25.225037 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.224916    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f1584e1-c962-4d28-8c23-7885eccc45f6-model-cache" pod "0f1584e1-c962-4d28-8c23-7885eccc45f6" (UID: "0f1584e1-c962-4d28-8c23-7885eccc45f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:01:25.225442 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.225421    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e697407f-b6fb-45d7-bc7f-743879e389c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-224496da-f7xt7\" (UID: \"e697407f-b6fb-45d7-bc7f-743879e389c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-224496da-f7xt7"
Nov 03 09:01:25.226352 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.226330    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f1584e1-c962-4d28-8c23-7885eccc45f6-dshm" pod "0f1584e1-c962-4d28-8c23-7885eccc45f6" (UID: "0f1584e1-c962-4d28-8c23-7885eccc45f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:01:25.226549 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.226522    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f1584e1-c962-4d28-8c23-7885eccc45f6-tls-certs" pod "0f1584e1-c962-4d28-8c23-7885eccc45f6" (UID: "0f1584e1-c962-4d28-8c23-7885eccc45f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:01:25.227114 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.227096    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e697407f-b6fb-45d7-bc7f-743879e389c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-224496da-f7xt7\" (UID: \"e697407f-b6fb-45d7-bc7f-743879e389c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-224496da-f7xt7"
Nov 03 09:01:25.227382 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.227368    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f1584e1-c962-4d28-8c23-7885eccc45f6-kube-api-access-k9d7v" pod "0f1584e1-c962-4d28-8c23-7885eccc45f6" (UID: "0f1584e1-c962-4d28-8c23-7885eccc45f6"). InnerVolumeSpecName "kube-api-access-k9d7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:01:25.284163 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.284099    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpnq9\" (UniqueName: \"kubernetes.io/projected/e697407f-b6fb-45d7-bc7f-743879e389c5-kube-api-access-fpnq9\") pod \"maas-api-revoke-sub-224496da-f7xt7\" (UID: \"e697407f-b6fb-45d7-bc7f-743879e389c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-224496da-f7xt7"
Nov 03 09:01:25.303442 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.303421    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-224496da-f7xt7"
Nov 03 09:01:25.326451 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.326424    4473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f1584e1-c962-4d28-8c23-7885eccc45f6-tmp-dir\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:01:25.326451 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.326452    4473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f1584e1-c962-4d28-8c23-7885eccc45f6-dshm\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:01:25.326593 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.326464    4473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f1584e1-c962-4d28-8c23-7885eccc45f6-model-cache\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:01:25.326593 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.326478    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9d7v\" (UniqueName: \"kubernetes.io/projected/0f1584e1-c962-4d28-8c23-7885eccc45f6-kube-api-access-k9d7v\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:01:25.326593 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.326491    4473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f1584e1-c962-4d28-8c23-7885eccc45f6-tls-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:01:25.680402 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.680378    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-224496da-f7xt7"]
Nov 03 09:01:25.680937 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:01:25.680910    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode697407f_b6fb_45d7_bc7f_743879e389c5.slice/crio-4c5d795cafacffcd940d10ff79edcd73b6cce1c7ef632b80fabca9df4eabb5a7 WatchSource:0}: Error finding container 4c5d795cafacffcd940d10ff79edcd73b6cce1c7ef632b80fabca9df4eabb5a7: Status 404 returned error can't find the container with id 4c5d795cafacffcd940d10ff79edcd73b6cce1c7ef632b80fabca9df4eabb5a7
Nov 03 09:01:25.752413 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.752385    4473 generic.go:356] "Generic (PLEG): container finished" podID="0f1584e1-c962-4d28-8c23-7885eccc45f6" containerID="4e2536c542b85ca9b67e7f27b5d47010600c2ede499eb0447ae78ca875ffae31" exitCode=0
Nov 03 09:01:25.752523 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.752441    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j"
Nov 03 09:01:25.752523 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.752477    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j" event={"ID":"0f1584e1-c962-4d28-8c23-7885eccc45f6","Type":"ContainerDied","Data":"4e2536c542b85ca9b67e7f27b5d47010600c2ede499eb0447ae78ca875ffae31"}
Nov 03 09:01:25.752643 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.752522    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-910d50/e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j" event={"ID":"0f1584e1-c962-4d28-8c23-7885eccc45f6","Type":"ContainerDied","Data":"ad7bd2e66e2257f4bb227a97ae8ae81567b82e99e3f9d0dd331f6171a69a2dfd"}
Nov 03 09:01:25.752643 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.752545    4473 scope.go:122] "RemoveContainer" containerID="4e2536c542b85ca9b67e7f27b5d47010600c2ede499eb0447ae78ca875ffae31"
Nov 03 09:01:25.754745 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.754720    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn" event={"ID":"035b0461-2dfa-42f4-8ea7-2d1911cddca9","Type":"ContainerStarted","Data":"66b8d0a5f7fd81910c4dc73e8c6cc0836b43a2cb8c41b98c1207c16860005d9d"}
Nov 03 09:01:25.754860 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.754762    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn" event={"ID":"035b0461-2dfa-42f4-8ea7-2d1911cddca9","Type":"ContainerStarted","Data":"b53cbffc02d3c5c1d8d6160475d10f8430c8986fdd652c58a1c41f4878121d43"}
Nov 03 09:01:25.756529 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.756506    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-224496da-f7xt7" event={"ID":"e697407f-b6fb-45d7-bc7f-743879e389c5","Type":"ContainerStarted","Data":"4c5d795cafacffcd940d10ff79edcd73b6cce1c7ef632b80fabca9df4eabb5a7"}
Nov 03 09:01:25.763232 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.763219    4473 scope.go:122] "RemoveContainer" containerID="4e2536c542b85ca9b67e7f27b5d47010600c2ede499eb0447ae78ca875ffae31"
Nov 03 09:01:25.763470 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:01:25.763454    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e2536c542b85ca9b67e7f27b5d47010600c2ede499eb0447ae78ca875ffae31\": container with ID starting with 4e2536c542b85ca9b67e7f27b5d47010600c2ede499eb0447ae78ca875ffae31 not found: ID does not exist" containerID="4e2536c542b85ca9b67e7f27b5d47010600c2ede499eb0447ae78ca875ffae31"
Nov 03 09:01:25.763505 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.763478    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e2536c542b85ca9b67e7f27b5d47010600c2ede499eb0447ae78ca875ffae31"} err="failed to get container status \"4e2536c542b85ca9b67e7f27b5d47010600c2ede499eb0447ae78ca875ffae31\": rpc error: code = NotFound desc = could not find container \"4e2536c542b85ca9b67e7f27b5d47010600c2ede499eb0447ae78ca875ffae31\": container with ID starting with 4e2536c542b85ca9b67e7f27b5d47010600c2ede499eb0447ae78ca875ffae31 not found: ID does not exist"
Nov 03 09:01:25.881106 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.881076    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63d7d166-wrnf4"]
Nov 03 09:01:25.881541 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.881528    4473 cpu_manager.go:414] "containerMap: removing container" podUID="0f1584e1-c962-4d28-8c23-7885eccc45f6" containerName="main"
Nov 03 09:01:25.881591 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.881542    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f1584e1-c962-4d28-8c23-7885eccc45f6" containerName="main"
Nov 03 09:01:25.881643 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.881633    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f1584e1-c962-4d28-8c23-7885eccc45f6" containerName="main"
Nov 03 09:01:25.886004 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.885988    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d7d166-wrnf4"
Nov 03 09:01:25.922586 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.922554    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-910d50/e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j"]
Nov 03 09:01:25.926702 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.926676    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-910d50/e2e-conflict-a-54a1bbf9-kserve-645996c4b5-5cz5j"]
Nov 03 09:01:25.933198 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.933175    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee74596e-1d41-4c5e-a156-b3655afab343-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63d7d166-wrnf4\" (UID: \"ee74596e-1d41-4c5e-a156-b3655afab343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d7d166-wrnf4"
Nov 03 09:01:25.933327 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.933310    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsnk7\" (UniqueName: \"kubernetes.io/projected/ee74596e-1d41-4c5e-a156-b3655afab343-kube-api-access-dsnk7\") pod \"maas-api-revoke-sub-63d7d166-wrnf4\" (UID: \"ee74596e-1d41-4c5e-a156-b3655afab343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d7d166-wrnf4"
Nov 03 09:01:25.933451 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.933428    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee74596e-1d41-4c5e-a156-b3655afab343-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63d7d166-wrnf4\" (UID: \"ee74596e-1d41-4c5e-a156-b3655afab343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d7d166-wrnf4"
Nov 03 09:01:25.942938 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:25.942911    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63d7d166-wrnf4"]
Nov 03 09:01:26.034350 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:26.034327    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee74596e-1d41-4c5e-a156-b3655afab343-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63d7d166-wrnf4\" (UID: \"ee74596e-1d41-4c5e-a156-b3655afab343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d7d166-wrnf4"
Nov 03 09:01:26.034447 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:26.034404    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsnk7\" (UniqueName: \"kubernetes.io/projected/ee74596e-1d41-4c5e-a156-b3655afab343-kube-api-access-dsnk7\") pod \"maas-api-revoke-sub-63d7d166-wrnf4\" (UID: \"ee74596e-1d41-4c5e-a156-b3655afab343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d7d166-wrnf4"
Nov 03 09:01:26.034447 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:26.034441    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee74596e-1d41-4c5e-a156-b3655afab343-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63d7d166-wrnf4\" (UID: \"ee74596e-1d41-4c5e-a156-b3655afab343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d7d166-wrnf4"
Nov 03 09:01:26.035121 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:26.035098    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee74596e-1d41-4c5e-a156-b3655afab343-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63d7d166-wrnf4\" (UID: \"ee74596e-1d41-4c5e-a156-b3655afab343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d7d166-wrnf4"
Nov 03 09:01:26.036562 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:26.036540    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee74596e-1d41-4c5e-a156-b3655afab343-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63d7d166-wrnf4\" (UID: \"ee74596e-1d41-4c5e-a156-b3655afab343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d7d166-wrnf4"
Nov 03 09:01:26.041613 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:26.041573    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn" podStartSLOduration=2.041561722 podStartE2EDuration="2.041561722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:01:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:01:26.039266154 +0000 UTC m=+1310.308329607" watchObservedRunningTime="2026-11-03 09:01:26.041561722 +0000 UTC m=+1310.310625153"
Nov 03 09:01:26.094012 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:26.093984    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsnk7\" (UniqueName: \"kubernetes.io/projected/ee74596e-1d41-4c5e-a156-b3655afab343-kube-api-access-dsnk7\") pod \"maas-api-revoke-sub-63d7d166-wrnf4\" (UID: \"ee74596e-1d41-4c5e-a156-b3655afab343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d7d166-wrnf4"
Nov 03 09:01:26.202230 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:26.202147    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d7d166-wrnf4"
Nov 03 09:01:26.589070 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:26.589045    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63d7d166-wrnf4"]
Nov 03 09:01:26.590199 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:01:26.590169    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee74596e_1d41_4c5e_a156_b3655afab343.slice/crio-e32fac1eef68c0ce0b18350b42e22e430bcb1dfa80202ad80d39095b3340f00c WatchSource:0}: Error finding container e32fac1eef68c0ce0b18350b42e22e430bcb1dfa80202ad80d39095b3340f00c: Status 404 returned error can't find the container with id e32fac1eef68c0ce0b18350b42e22e430bcb1dfa80202ad80d39095b3340f00c
Nov 03 09:01:26.715023 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:26.714998    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn"
Nov 03 09:01:26.764856 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:26.764824    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d7d166-wrnf4" event={"ID":"ee74596e-1d41-4c5e-a156-b3655afab343","Type":"ContainerStarted","Data":"e32fac1eef68c0ce0b18350b42e22e430bcb1dfa80202ad80d39095b3340f00c"}
Nov 03 09:01:26.766400 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:26.766373    4473 generic.go:356] "Generic (PLEG): container finished" podID="e697407f-b6fb-45d7-bc7f-743879e389c5" containerID="d366fa85856c8ac7d0cfb90d17669ccd9bb9acc83073daffb9c4ea8ead046c37" exitCode=0
Nov 03 09:01:26.766491 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:26.766451    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-224496da-f7xt7" event={"ID":"e697407f-b6fb-45d7-bc7f-743879e389c5","Type":"ContainerDied","Data":"d366fa85856c8ac7d0cfb90d17669ccd9bb9acc83073daffb9c4ea8ead046c37"}
Nov 03 09:01:27.063727 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:27.063693    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a3eba79-h9hvz"]
Nov 03 09:01:27.067760 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:27.067736    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a3eba79-h9hvz"]
Nov 03 09:01:27.774156 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:27.774071    4473 generic.go:356] "Generic (PLEG): container finished" podID="ee74596e-1d41-4c5e-a156-b3655afab343" containerID="fae32803d23e9a04a827800091182dd81863f1692e47e3cf6f61591af563f797" exitCode=0
Nov 03 09:01:27.774317 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:27.774154    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d7d166-wrnf4" event={"ID":"ee74596e-1d41-4c5e-a156-b3655afab343","Type":"ContainerDied","Data":"fae32803d23e9a04a827800091182dd81863f1692e47e3cf6f61591af563f797"}
Nov 03 09:01:27.872495 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:27.872459    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f1584e1-c962-4d28-8c23-7885eccc45f6" path="/var/lib/kubelet/pods/0f1584e1-c962-4d28-8c23-7885eccc45f6/volumes"
Nov 03 09:01:27.872928 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:27.872910    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f8cb3cb-6a02-463d-a8c3-cacbf5449af5" path="/var/lib/kubelet/pods/3f8cb3cb-6a02-463d-a8c3-cacbf5449af5/volumes"
Nov 03 09:01:27.921970 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:27.921919    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-224496da-f7xt7"
Nov 03 09:01:27.958720 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:27.958696    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e697407f-b6fb-45d7-bc7f-743879e389c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e697407f-b6fb-45d7-bc7f-743879e389c5-maas-api-xks-ca\") pod \"e697407f-b6fb-45d7-bc7f-743879e389c5\" (UID: \"e697407f-b6fb-45d7-bc7f-743879e389c5\") "
Nov 03 09:01:27.958845 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:27.958725    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e697407f-b6fb-45d7-bc7f-743879e389c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e697407f-b6fb-45d7-bc7f-743879e389c5-maas-api-openshift-ca\") pod \"e697407f-b6fb-45d7-bc7f-743879e389c5\" (UID: \"e697407f-b6fb-45d7-bc7f-743879e389c5\") "
Nov 03 09:01:27.958845 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:27.958789    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e697407f-b6fb-45d7-bc7f-743879e389c5-kube-api-access-fpnq9\" (UniqueName: \"kubernetes.io/projected/e697407f-b6fb-45d7-bc7f-743879e389c5-kube-api-access-fpnq9\") pod \"e697407f-b6fb-45d7-bc7f-743879e389c5\" (UID: \"e697407f-b6fb-45d7-bc7f-743879e389c5\") "
Nov 03 09:01:27.959127 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:27.959099    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e697407f-b6fb-45d7-bc7f-743879e389c5-maas-api-openshift-ca" pod "e697407f-b6fb-45d7-bc7f-743879e389c5" (UID: "e697407f-b6fb-45d7-bc7f-743879e389c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:01:27.959323 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:27.959302    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e697407f-b6fb-45d7-bc7f-743879e389c5-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:01:27.961020 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:27.960995    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e697407f-b6fb-45d7-bc7f-743879e389c5-maas-api-xks-ca" pod "e697407f-b6fb-45d7-bc7f-743879e389c5" (UID: "e697407f-b6fb-45d7-bc7f-743879e389c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:01:27.961098 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:27.961042    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e697407f-b6fb-45d7-bc7f-743879e389c5-kube-api-access-fpnq9" pod "e697407f-b6fb-45d7-bc7f-743879e389c5" (UID: "e697407f-b6fb-45d7-bc7f-743879e389c5"). InnerVolumeSpecName "kube-api-access-fpnq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:01:28.060610 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:28.060547    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e697407f-b6fb-45d7-bc7f-743879e389c5-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:01:28.060610 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:28.060574    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpnq9\" (UniqueName: \"kubernetes.io/projected/e697407f-b6fb-45d7-bc7f-743879e389c5-kube-api-access-fpnq9\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:01:28.781333 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:28.781289    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-224496da-f7xt7" event={"ID":"e697407f-b6fb-45d7-bc7f-743879e389c5","Type":"ContainerDied","Data":"4c5d795cafacffcd940d10ff79edcd73b6cce1c7ef632b80fabca9df4eabb5a7"}
Nov 03 09:01:28.781333 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:28.781317    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-224496da-f7xt7"
Nov 03 09:01:28.782029 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:28.781353    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c5d795cafacffcd940d10ff79edcd73b6cce1c7ef632b80fabca9df4eabb5a7"
Nov 03 09:01:28.925266 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:28.925240    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d7d166-wrnf4"
Nov 03 09:01:28.968506 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:28.968470    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee74596e-1d41-4c5e-a156-b3655afab343-kube-api-access-dsnk7\" (UniqueName: \"kubernetes.io/projected/ee74596e-1d41-4c5e-a156-b3655afab343-kube-api-access-dsnk7\") pod \"ee74596e-1d41-4c5e-a156-b3655afab343\" (UID: \"ee74596e-1d41-4c5e-a156-b3655afab343\") "
Nov 03 09:01:28.968652 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:28.968545    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee74596e-1d41-4c5e-a156-b3655afab343-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee74596e-1d41-4c5e-a156-b3655afab343-maas-api-openshift-ca\") pod \"ee74596e-1d41-4c5e-a156-b3655afab343\" (UID: \"ee74596e-1d41-4c5e-a156-b3655afab343\") "
Nov 03 09:01:28.968652 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:28.968565    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee74596e-1d41-4c5e-a156-b3655afab343-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee74596e-1d41-4c5e-a156-b3655afab343-maas-api-xks-ca\") pod \"ee74596e-1d41-4c5e-a156-b3655afab343\" (UID: \"ee74596e-1d41-4c5e-a156-b3655afab343\") "
Nov 03 09:01:28.968925 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:28.968897    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee74596e-1d41-4c5e-a156-b3655afab343-maas-api-openshift-ca" pod "ee74596e-1d41-4c5e-a156-b3655afab343" (UID: "ee74596e-1d41-4c5e-a156-b3655afab343"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:01:28.969103 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:28.969085    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee74596e-1d41-4c5e-a156-b3655afab343-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:01:28.971280 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:28.971261    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee74596e-1d41-4c5e-a156-b3655afab343-maas-api-xks-ca" pod "ee74596e-1d41-4c5e-a156-b3655afab343" (UID: "ee74596e-1d41-4c5e-a156-b3655afab343"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:01:28.971362 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:28.971285    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee74596e-1d41-4c5e-a156-b3655afab343-kube-api-access-dsnk7" pod "ee74596e-1d41-4c5e-a156-b3655afab343" (UID: "ee74596e-1d41-4c5e-a156-b3655afab343"). InnerVolumeSpecName "kube-api-access-dsnk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:01:29.069810 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:29.069747    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsnk7\" (UniqueName: \"kubernetes.io/projected/ee74596e-1d41-4c5e-a156-b3655afab343-kube-api-access-dsnk7\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:01:29.069810 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:29.069773    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee74596e-1d41-4c5e-a156-b3655afab343-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:01:29.790146 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:29.790118    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d7d166-wrnf4"
Nov 03 09:01:29.790551 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:29.790149    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63d7d166-wrnf4" event={"ID":"ee74596e-1d41-4c5e-a156-b3655afab343","Type":"ContainerDied","Data":"e32fac1eef68c0ce0b18350b42e22e430bcb1dfa80202ad80d39095b3340f00c"}
Nov 03 09:01:29.790551 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:29.790210    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e32fac1eef68c0ce0b18350b42e22e430bcb1dfa80202ad80d39095b3340f00c"
Nov 03 09:01:31.422022 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:31.421983    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a40cf0f3-6c6z8"]
Nov 03 09:01:31.422672 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:31.422649    4473 cpu_manager.go:414] "containerMap: removing container" podUID="e697407f-b6fb-45d7-bc7f-743879e389c5" containerName="revoke-keys"
Nov 03 09:01:31.422764 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:31.422675    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e697407f-b6fb-45d7-bc7f-743879e389c5" containerName="revoke-keys"
Nov 03 09:01:31.422764 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:31.422724    4473 cpu_manager.go:414] "containerMap: removing container" podUID="ee74596e-1d41-4c5e-a156-b3655afab343" containerName="revoke-keys"
Nov 03 09:01:31.422764 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:31.422732    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee74596e-1d41-4c5e-a156-b3655afab343" containerName="revoke-keys"
Nov 03 09:01:31.422915 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:31.422865    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="e697407f-b6fb-45d7-bc7f-743879e389c5" containerName="revoke-keys"
Nov 03 09:01:31.422915 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:31.422886    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee74596e-1d41-4c5e-a156-b3655afab343" containerName="revoke-keys"
Nov 03 09:01:31.426256 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:31.426236    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40cf0f3-6c6z8"
Nov 03 09:01:31.482971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:31.482929    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a40cf0f3-6c6z8"]
Nov 03 09:01:31.510011 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:31.509984    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qktnx\" (UniqueName: \"kubernetes.io/projected/92df7d0e-c794-4e54-b75f-0d238d209d16-kube-api-access-qktnx\") pod \"maas-api-revoke-sub-a40cf0f3-6c6z8\" (UID: \"92df7d0e-c794-4e54-b75f-0d238d209d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40cf0f3-6c6z8"
Nov 03 09:01:31.510167 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:31.510149    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92df7d0e-c794-4e54-b75f-0d238d209d16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a40cf0f3-6c6z8\" (UID: \"92df7d0e-c794-4e54-b75f-0d238d209d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40cf0f3-6c6z8"
Nov 03 09:01:31.510269 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:31.510249    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92df7d0e-c794-4e54-b75f-0d238d209d16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a40cf0f3-6c6z8\" (UID: \"92df7d0e-c794-4e54-b75f-0d238d209d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40cf0f3-6c6z8"
Nov 03 09:01:31.530076 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:31.530051    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:01:31.530187 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:31.530054    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:01:31.530187 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:31.530061    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:01:31.530187 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:31.530073    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:01:31.611874 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:31.611841    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qktnx\" (UniqueName: \"kubernetes.io/projected/92df7d0e-c794-4e54-b75f-0d238d209d16-kube-api-access-qktnx\") pod \"maas-api-revoke-sub-a40cf0f3-6c6z8\" (UID: \"92df7d0e-c794-4e54-b75f-0d238d209d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40cf0f3-6c6z8"
Nov 03 09:01:31.612084 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:31.612048    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92df7d0e-c794-4e54-b75f-0d238d209d16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a40cf0f3-6c6z8\" (UID: \"92df7d0e-c794-4e54-b75f-0d238d209d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40cf0f3-6c6z8"
Nov 03 09:01:31.612157 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:31.612121    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92df7d0e-c794-4e54-b75f-0d238d209d16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a40cf0f3-6c6z8\" (UID: \"92df7d0e-c794-4e54-b75f-0d238d209d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40cf0f3-6c6z8"
Nov 03 09:01:31.613042 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:31.612974    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92df7d0e-c794-4e54-b75f-0d238d209d16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a40cf0f3-6c6z8\" (UID: \"92df7d0e-c794-4e54-b75f-0d238d209d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40cf0f3-6c6z8"
Nov 03 09:01:31.631422 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:31.627498    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92df7d0e-c794-4e54-b75f-0d238d209d16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a40cf0f3-6c6z8\" (UID: \"92df7d0e-c794-4e54-b75f-0d238d209d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40cf0f3-6c6z8"
Nov 03 09:01:31.681599 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:31.681527    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qktnx\" (UniqueName: \"kubernetes.io/projected/92df7d0e-c794-4e54-b75f-0d238d209d16-kube-api-access-qktnx\") pod \"maas-api-revoke-sub-a40cf0f3-6c6z8\" (UID: \"92df7d0e-c794-4e54-b75f-0d238d209d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40cf0f3-6c6z8"
Nov 03 09:01:31.740191 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:31.740150    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40cf0f3-6c6z8"
Nov 03 09:01:32.131658 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.131632    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a40cf0f3-6c6z8"]
Nov 03 09:01:32.132767 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:01:32.132740    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92df7d0e_c794_4e54_b75f_0d238d209d16.slice/crio-d44884b7e5c88d3c8d48139b955d637862f86ff1319355213b1d9a4be05ddc40 WatchSource:0}: Error finding container d44884b7e5c88d3c8d48139b955d637862f86ff1319355213b1d9a4be05ddc40: Status 404 returned error can't find the container with id d44884b7e5c88d3c8d48139b955d637862f86ff1319355213b1d9a4be05ddc40
Nov 03 09:01:32.617153 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.617113    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"]
Nov 03 09:01:32.621154 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.621134    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"
Nov 03 09:01:32.667701 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.667673    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"]
Nov 03 09:01:32.671334 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.671318    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"
Nov 03 09:01:32.722238 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.722213    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"]
Nov 03 09:01:32.731867 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.731840    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1febda0d-e4ac-436c-863e-902751af88ca-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4\" (UID: \"1febda0d-e4ac-436c-863e-902751af88ca\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"
Nov 03 09:01:32.731995 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.731873    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc159b5a-6ae2-4f09-a74d-1ff15927d692-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr\" (UID: \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"
Nov 03 09:01:32.731995 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.731984    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1febda0d-e4ac-436c-863e-902751af88ca-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4\" (UID: \"1febda0d-e4ac-436c-863e-902751af88ca\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"
Nov 03 09:01:32.732114 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.732024    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc159b5a-6ae2-4f09-a74d-1ff15927d692-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr\" (UID: \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"
Nov 03 09:01:32.732114 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.732046    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1febda0d-e4ac-436c-863e-902751af88ca-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4\" (UID: \"1febda0d-e4ac-436c-863e-902751af88ca\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"
Nov 03 09:01:32.732223 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.732114    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1febda0d-e4ac-436c-863e-902751af88ca-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4\" (UID: \"1febda0d-e4ac-436c-863e-902751af88ca\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"
Nov 03 09:01:32.732223 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.732148    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc159b5a-6ae2-4f09-a74d-1ff15927d692-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr\" (UID: \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"
Nov 03 09:01:32.732223 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.732189    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1febda0d-e4ac-436c-863e-902751af88ca-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4\" (UID: \"1febda0d-e4ac-436c-863e-902751af88ca\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"
Nov 03 09:01:32.732342 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.732257    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1febda0d-e4ac-436c-863e-902751af88ca-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4\" (UID: \"1febda0d-e4ac-436c-863e-902751af88ca\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"
Nov 03 09:01:32.732342 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.732276    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc159b5a-6ae2-4f09-a74d-1ff15927d692-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr\" (UID: \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"
Nov 03 09:01:32.732342 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.732294    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vqbx\" (UniqueName: \"kubernetes.io/projected/1febda0d-e4ac-436c-863e-902751af88ca-kube-api-access-9vqbx\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4\" (UID: \"1febda0d-e4ac-436c-863e-902751af88ca\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"
Nov 03 09:01:32.732502 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.732351    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc159b5a-6ae2-4f09-a74d-1ff15927d692-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr\" (UID: \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"
Nov 03 09:01:32.732502 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.732432    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc159b5a-6ae2-4f09-a74d-1ff15927d692-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr\" (UID: \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"
Nov 03 09:01:32.732502 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.732465    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1febda0d-e4ac-436c-863e-902751af88ca-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4\" (UID: \"1febda0d-e4ac-436c-863e-902751af88ca\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"
Nov 03 09:01:32.732664 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.732567    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1febda0d-e4ac-436c-863e-902751af88ca-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4\" (UID: \"1febda0d-e4ac-436c-863e-902751af88ca\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"
Nov 03 09:01:32.732664 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.732593    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc159b5a-6ae2-4f09-a74d-1ff15927d692-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr\" (UID: \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"
Nov 03 09:01:32.732664 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.732647    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqc9j\" (UniqueName: \"kubernetes.io/projected/cc159b5a-6ae2-4f09-a74d-1ff15927d692-kube-api-access-hqc9j\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr\" (UID: \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"
Nov 03 09:01:32.732827 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.732676    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc159b5a-6ae2-4f09-a74d-1ff15927d692-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr\" (UID: \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"
Nov 03 09:01:32.782480 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.782405    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"]
Nov 03 09:01:32.813314 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.813268    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40cf0f3-6c6z8" event={"ID":"92df7d0e-c794-4e54-b75f-0d238d209d16","Type":"ContainerStarted","Data":"be7e763f0e6042b8fc634aca61f7dcf8d7ddd4bcdf1be1539b6bc9c22ed329c2"}
Nov 03 09:01:32.813472 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.813344    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40cf0f3-6c6z8" event={"ID":"92df7d0e-c794-4e54-b75f-0d238d209d16","Type":"ContainerStarted","Data":"d44884b7e5c88d3c8d48139b955d637862f86ff1319355213b1d9a4be05ddc40"}
Nov 03 09:01:32.833786 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.833761    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc159b5a-6ae2-4f09-a74d-1ff15927d692-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr\" (UID: \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"
Nov 03 09:01:32.833958 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.833795    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1febda0d-e4ac-436c-863e-902751af88ca-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4\" (UID: \"1febda0d-e4ac-436c-863e-902751af88ca\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"
Nov 03 09:01:32.833958 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.833812    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1febda0d-e4ac-436c-863e-902751af88ca-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4\" (UID: \"1febda0d-e4ac-436c-863e-902751af88ca\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"
Nov 03 09:01:32.833958 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.833827    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc159b5a-6ae2-4f09-a74d-1ff15927d692-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr\" (UID: \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"
Nov 03 09:01:32.833958 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.833850    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1febda0d-e4ac-436c-863e-902751af88ca-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4\" (UID: \"1febda0d-e4ac-436c-863e-902751af88ca\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"
Nov 03 09:01:32.833958 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.833872    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1febda0d-e4ac-436c-863e-902751af88ca-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4\" (UID: \"1febda0d-e4ac-436c-863e-902751af88ca\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"
Nov 03 09:01:32.833958 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.833892    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc159b5a-6ae2-4f09-a74d-1ff15927d692-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr\" (UID: \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"
Nov 03 09:01:32.833958 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.833916    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vqbx\" (UniqueName: \"kubernetes.io/projected/1febda0d-e4ac-436c-863e-902751af88ca-kube-api-access-9vqbx\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4\" (UID: \"1febda0d-e4ac-436c-863e-902751af88ca\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"
Nov 03 09:01:32.834905 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.833967    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc159b5a-6ae2-4f09-a74d-1ff15927d692-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr\" (UID: \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"
Nov 03 09:01:32.834905 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.834006    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc159b5a-6ae2-4f09-a74d-1ff15927d692-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr\" (UID: \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"
Nov 03 09:01:32.834905 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.834033    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1febda0d-e4ac-436c-863e-902751af88ca-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4\" (UID: \"1febda0d-e4ac-436c-863e-902751af88ca\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"
Nov 03 09:01:32.834905 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.834090    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1febda0d-e4ac-436c-863e-902751af88ca-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4\" (UID: \"1febda0d-e4ac-436c-863e-902751af88ca\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"
Nov 03 09:01:32.834905 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.834126    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc159b5a-6ae2-4f09-a74d-1ff15927d692-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr\" (UID: \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"
Nov 03 09:01:32.834905 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.834163    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqc9j\" (UniqueName: \"kubernetes.io/projected/cc159b5a-6ae2-4f09-a74d-1ff15927d692-kube-api-access-hqc9j\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr\" (UID: \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"
Nov 03 09:01:32.834905 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.834191    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc159b5a-6ae2-4f09-a74d-1ff15927d692-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr\" (UID: \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"
Nov 03 09:01:32.834905 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.834293    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1febda0d-e4ac-436c-863e-902751af88ca-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4\" (UID: \"1febda0d-e4ac-436c-863e-902751af88ca\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"
Nov 03 09:01:32.834905 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.834318    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc159b5a-6ae2-4f09-a74d-1ff15927d692-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr\" (UID: \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"
Nov 03 09:01:32.834905 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.834388    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1febda0d-e4ac-436c-863e-902751af88ca-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4\" (UID: \"1febda0d-e4ac-436c-863e-902751af88ca\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"
Nov 03 09:01:32.834905 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.834395    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc159b5a-6ae2-4f09-a74d-1ff15927d692-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr\" (UID: \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"
Nov 03 09:01:32.834905 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.834583    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc159b5a-6ae2-4f09-a74d-1ff15927d692-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr\" (UID: \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"
Nov 03 09:01:32.834905 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.834594    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1febda0d-e4ac-436c-863e-902751af88ca-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4\" (UID: \"1febda0d-e4ac-436c-863e-902751af88ca\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"
Nov 03 09:01:32.835563 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.835175    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1febda0d-e4ac-436c-863e-902751af88ca-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4\" (UID: \"1febda0d-e4ac-436c-863e-902751af88ca\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"
Nov 03 09:01:32.835563 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.835191    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1febda0d-e4ac-436c-863e-902751af88ca-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4\" (UID: \"1febda0d-e4ac-436c-863e-902751af88ca\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"
Nov 03 09:01:32.835563 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.835393    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1febda0d-e4ac-436c-863e-902751af88ca-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4\" (UID: \"1febda0d-e4ac-436c-863e-902751af88ca\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"
Nov 03 09:01:32.835728 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.835566    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1febda0d-e4ac-436c-863e-902751af88ca-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4\" (UID: \"1febda0d-e4ac-436c-863e-902751af88ca\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"
Nov 03 09:01:32.835728 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.835615    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc159b5a-6ae2-4f09-a74d-1ff15927d692-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr\" (UID: \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"
Nov 03 09:01:32.836158 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.836132    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc159b5a-6ae2-4f09-a74d-1ff15927d692-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr\" (UID: \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"
Nov 03 09:01:32.836263 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.836195    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc159b5a-6ae2-4f09-a74d-1ff15927d692-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr\" (UID: \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"
Nov 03 09:01:32.837535 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.837303    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc159b5a-6ae2-4f09-a74d-1ff15927d692-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr\" (UID: \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"
Nov 03 09:01:32.838393 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.837910    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1febda0d-e4ac-436c-863e-902751af88ca-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4\" (UID: \"1febda0d-e4ac-436c-863e-902751af88ca\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"
Nov 03 09:01:32.838881 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.838839    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc159b5a-6ae2-4f09-a74d-1ff15927d692-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr\" (UID: \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"
Nov 03 09:01:32.840155 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.840132    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1febda0d-e4ac-436c-863e-902751af88ca-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4\" (UID: \"1febda0d-e4ac-436c-863e-902751af88ca\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"
Nov 03 09:01:32.893834 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.893809    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc159b5a-6ae2-4f09-a74d-1ff15927d692-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr\" (UID: \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"
Nov 03 09:01:32.893984 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.893915    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1febda0d-e4ac-436c-863e-902751af88ca-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4\" (UID: \"1febda0d-e4ac-436c-863e-902751af88ca\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"
Nov 03 09:01:32.893984 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.893974    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqc9j\" (UniqueName: \"kubernetes.io/projected/cc159b5a-6ae2-4f09-a74d-1ff15927d692-kube-api-access-hqc9j\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr\" (UID: \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"
Nov 03 09:01:32.894070 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.894055    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vqbx\" (UniqueName: \"kubernetes.io/projected/1febda0d-e4ac-436c-863e-902751af88ca-kube-api-access-9vqbx\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4\" (UID: \"1febda0d-e4ac-436c-863e-902751af88ca\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"
Nov 03 09:01:32.938668 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.938604    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"
Nov 03 09:01:32.980297 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.979271    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40cf0f3-6c6z8" podStartSLOduration=1.979250562 podStartE2EDuration="1.979250562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:01:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:01:32.976527005 +0000 UTC m=+1317.245590479" watchObservedRunningTime="2026-11-03 09:01:32.979250562 +0000 UTC m=+1317.248313997"
Nov 03 09:01:32.983307 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:32.983136    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"
Nov 03 09:01:33.376747 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:33.376719    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"]
Nov 03 09:01:33.377646 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:01:33.377610    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1febda0d_e4ac_436c_863e_902751af88ca.slice/crio-c05ad90193fed68f4013b3f4b1d6bd6ad9415d051c536e35ac9250bf20ee45c9 WatchSource:0}: Error finding container c05ad90193fed68f4013b3f4b1d6bd6ad9415d051c536e35ac9250bf20ee45c9: Status 404 returned error can't find the container with id c05ad90193fed68f4013b3f4b1d6bd6ad9415d051c536e35ac9250bf20ee45c9
Nov 03 09:01:33.380022 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:33.379988    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:01:33.380114 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:33.380072    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:01:33.380158 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:33.380140    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:01:33.384866 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:33.384840    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"]
Nov 03 09:01:33.420302 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:01:33.420269    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc159b5a_6ae2_4f09_a74d_1ff15927d692.slice/crio-791621d0a229dce644637e5ffa2a555b0548fda86ad062165bcc77064f661f0a WatchSource:0}: Error finding container 791621d0a229dce644637e5ffa2a555b0548fda86ad062165bcc77064f661f0a: Status 404 returned error can't find the container with id 791621d0a229dce644637e5ffa2a555b0548fda86ad062165bcc77064f661f0a
Nov 03 09:01:33.422661 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:33.422635    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:01:33.422749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:33.422690    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:01:33.422790 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:33.422749    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:01:33.822610 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:33.822553    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4" event={"ID":"1febda0d-e4ac-436c-863e-902751af88ca","Type":"ContainerStarted","Data":"171c40d5e73be093a52718481f53fce86703c829f1475be01709f718c6cef14d"}
Nov 03 09:01:33.823135 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:33.822692    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4" event={"ID":"1febda0d-e4ac-436c-863e-902751af88ca","Type":"ContainerStarted","Data":"c05ad90193fed68f4013b3f4b1d6bd6ad9415d051c536e35ac9250bf20ee45c9"}
Nov 03 09:01:33.824525 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:33.824494    4473 generic.go:356] "Generic (PLEG): container finished" podID="92df7d0e-c794-4e54-b75f-0d238d209d16" containerID="be7e763f0e6042b8fc634aca61f7dcf8d7ddd4bcdf1be1539b6bc9c22ed329c2" exitCode=0
Nov 03 09:01:33.824667 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:33.824581    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40cf0f3-6c6z8" event={"ID":"92df7d0e-c794-4e54-b75f-0d238d209d16","Type":"ContainerDied","Data":"be7e763f0e6042b8fc634aca61f7dcf8d7ddd4bcdf1be1539b6bc9c22ed329c2"}
Nov 03 09:01:33.826454 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:33.826426    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr" event={"ID":"cc159b5a-6ae2-4f09-a74d-1ff15927d692","Type":"ContainerStarted","Data":"cd9aeb3769f2c937b77f31625d243052b7d238938466e2a983af477c93f9141b"}
Nov 03 09:01:33.826601 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:33.826481    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr" event={"ID":"cc159b5a-6ae2-4f09-a74d-1ff15927d692","Type":"ContainerStarted","Data":"791621d0a229dce644637e5ffa2a555b0548fda86ad062165bcc77064f661f0a"}
Nov 03 09:01:33.939121 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:33.939009    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"
Nov 03 09:01:33.987099 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:33.987057    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"
Nov 03 09:01:34.073580 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:34.073521    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr" podStartSLOduration=2.073506518 podStartE2EDuration="2.073506518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:01:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:01:34.073205474 +0000 UTC m=+1318.342268905" watchObservedRunningTime="2026-11-03 09:01:34.073506518 +0000 UTC m=+1318.342569954"
Nov 03 09:01:34.073971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:34.073770    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4" podStartSLOduration=2.073761879 podStartE2EDuration="2.073761879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:01:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:01:33.954232178 +0000 UTC m=+1318.223295659" watchObservedRunningTime="2026-11-03 09:01:34.073761879 +0000 UTC m=+1318.342825313"
Nov 03 09:01:34.715249 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:34.715221    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn"
Nov 03 09:01:34.776614 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:34.776585    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn"
Nov 03 09:01:34.885708 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:34.885676    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn"
Nov 03 09:01:34.950970 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:34.949225    4473 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 09:01:34.950970 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:34.949293    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4" podUID="1febda0d-e4ac-436c-863e-902751af88ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 09:01:34.985120 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:34.985085    4473 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 09:01:34.985378 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:34.985138    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr" podUID="cc159b5a-6ae2-4f09-a74d-1ff15927d692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 09:01:35.106459 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:35.106438    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40cf0f3-6c6z8"
Nov 03 09:01:35.179964 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:35.179472    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92df7d0e-c794-4e54-b75f-0d238d209d16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92df7d0e-c794-4e54-b75f-0d238d209d16-maas-api-openshift-ca\") pod \"92df7d0e-c794-4e54-b75f-0d238d209d16\" (UID: \"92df7d0e-c794-4e54-b75f-0d238d209d16\") "
Nov 03 09:01:35.179964 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:35.179512    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92df7d0e-c794-4e54-b75f-0d238d209d16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92df7d0e-c794-4e54-b75f-0d238d209d16-maas-api-xks-ca\") pod \"92df7d0e-c794-4e54-b75f-0d238d209d16\" (UID: \"92df7d0e-c794-4e54-b75f-0d238d209d16\") "
Nov 03 09:01:35.179964 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:35.179536    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92df7d0e-c794-4e54-b75f-0d238d209d16-kube-api-access-qktnx\" (UniqueName: \"kubernetes.io/projected/92df7d0e-c794-4e54-b75f-0d238d209d16-kube-api-access-qktnx\") pod \"92df7d0e-c794-4e54-b75f-0d238d209d16\" (UID: \"92df7d0e-c794-4e54-b75f-0d238d209d16\") "
Nov 03 09:01:35.179964 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:35.179917    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92df7d0e-c794-4e54-b75f-0d238d209d16-maas-api-openshift-ca" pod "92df7d0e-c794-4e54-b75f-0d238d209d16" (UID: "92df7d0e-c794-4e54-b75f-0d238d209d16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:01:35.182174 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:35.182054    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92df7d0e-c794-4e54-b75f-0d238d209d16-kube-api-access-qktnx" pod "92df7d0e-c794-4e54-b75f-0d238d209d16" (UID: "92df7d0e-c794-4e54-b75f-0d238d209d16"). InnerVolumeSpecName "kube-api-access-qktnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:01:35.182433 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:35.182402    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92df7d0e-c794-4e54-b75f-0d238d209d16-maas-api-xks-ca" pod "92df7d0e-c794-4e54-b75f-0d238d209d16" (UID: "92df7d0e-c794-4e54-b75f-0d238d209d16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:01:35.280844 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:35.280760    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92df7d0e-c794-4e54-b75f-0d238d209d16-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:01:35.280844 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:35.280792    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92df7d0e-c794-4e54-b75f-0d238d209d16-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:01:35.280844 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:35.280805    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qktnx\" (UniqueName: \"kubernetes.io/projected/92df7d0e-c794-4e54-b75f-0d238d209d16-kube-api-access-qktnx\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:01:35.877938 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:35.877908    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40cf0f3-6c6z8"
Nov 03 09:01:35.878211 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:35.878186    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40cf0f3-6c6z8" event={"ID":"92df7d0e-c794-4e54-b75f-0d238d209d16","Type":"ContainerDied","Data":"d44884b7e5c88d3c8d48139b955d637862f86ff1319355213b1d9a4be05ddc40"}
Nov 03 09:01:35.878350 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:35.878336    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d44884b7e5c88d3c8d48139b955d637862f86ff1319355213b1d9a4be05ddc40"
Nov 03 09:01:35.944973 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:35.944826    4473 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 09:01:35.944973 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:35.944883    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4" podUID="1febda0d-e4ac-436c-863e-902751af88ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 09:01:35.991861 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:35.983981    4473 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 09:01:35.991861 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:35.984040    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr" podUID="cc159b5a-6ae2-4f09-a74d-1ff15927d692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 09:01:36.144020 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:36.143801    4473 scope.go:122] "RemoveContainer" containerID="9e5d0c9d89c3d73a5f74d91f6952a2b3de64a931653f29b1ed0af16a72f54656"
Nov 03 09:01:36.175457 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:36.175439    4473 scope.go:122] "RemoveContainer" containerID="90044752bc2d8c52df0b0f0c3a9828cc984ad63ae631813016a29b5aeaf96f55"
Nov 03 09:01:36.940381 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:36.940340    4473 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 09:01:36.940642 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:36.940610    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4" podUID="1febda0d-e4ac-436c-863e-902751af88ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 09:01:36.984026 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:36.983996    4473 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 09:01:36.984378 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:36.984048    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr" podUID="cc159b5a-6ae2-4f09-a74d-1ff15927d692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 09:01:37.964971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:37.964049    4473 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 09:01:37.964971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:37.964114    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4" podUID="1febda0d-e4ac-436c-863e-902751af88ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 09:01:37.983784 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:37.983754    4473 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 09:01:37.983933 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:37.983805    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr" podUID="cc159b5a-6ae2-4f09-a74d-1ff15927d692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 09:01:38.940489 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:38.940453    4473 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 09:01:38.941175 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:38.941130    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4" podUID="1febda0d-e4ac-436c-863e-902751af88ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 09:01:38.983810 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:38.983783    4473 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 09:01:38.984029 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:38.983916    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr" podUID="cc159b5a-6ae2-4f09-a74d-1ff15927d692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 09:01:39.939618 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:39.939583    4473 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Nov 03 09:01:39.939822 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:39.939642    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4" podUID="1febda0d-e4ac-436c-863e-902751af88ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Nov 03 09:01:39.984273 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:39.984239    4473 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Nov 03 09:01:39.984762 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:39.984301    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr" podUID="cc159b5a-6ae2-4f09-a74d-1ff15927d692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Nov 03 09:01:40.943696 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:40.943666    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"
Nov 03 09:01:40.944078 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:40.944059    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"
Nov 03 09:01:40.944628 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:40.944612    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"
Nov 03 09:01:40.988084 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:40.988056    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"
Nov 03 09:01:40.988455 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:40.988368    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"
Nov 03 09:01:40.989208 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:40.989188    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"
Nov 03 09:01:42.100979 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.100933    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"]
Nov 03 09:01:42.101528 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.101511    4473 cpu_manager.go:414] "containerMap: removing container" podUID="92df7d0e-c794-4e54-b75f-0d238d209d16" containerName="revoke-keys"
Nov 03 09:01:42.101572 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.101530    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92df7d0e-c794-4e54-b75f-0d238d209d16" containerName="revoke-keys"
Nov 03 09:01:42.101640 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.101631    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="92df7d0e-c794-4e54-b75f-0d238d209d16" containerName="revoke-keys"
Nov 03 09:01:42.106370 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.106351    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"
Nov 03 09:01:42.220967 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.220617    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkrtg\" (UniqueName: \"kubernetes.io/projected/bddb43dc-284b-475c-8f28-d8b320be94b3-kube-api-access-xkrtg\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9\" (UID: \"bddb43dc-284b-475c-8f28-d8b320be94b3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"
Nov 03 09:01:42.220967 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.220715    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bddb43dc-284b-475c-8f28-d8b320be94b3-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9\" (UID: \"bddb43dc-284b-475c-8f28-d8b320be94b3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"
Nov 03 09:01:42.220967 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.220751    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bddb43dc-284b-475c-8f28-d8b320be94b3-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9\" (UID: \"bddb43dc-284b-475c-8f28-d8b320be94b3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"
Nov 03 09:01:42.220967 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.220784    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bddb43dc-284b-475c-8f28-d8b320be94b3-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9\" (UID: \"bddb43dc-284b-475c-8f28-d8b320be94b3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"
Nov 03 09:01:42.220967 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.220808    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bddb43dc-284b-475c-8f28-d8b320be94b3-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9\" (UID: \"bddb43dc-284b-475c-8f28-d8b320be94b3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"
Nov 03 09:01:42.220967 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.220867    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bddb43dc-284b-475c-8f28-d8b320be94b3-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9\" (UID: \"bddb43dc-284b-475c-8f28-d8b320be94b3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"
Nov 03 09:01:42.220967 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.220902    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bddb43dc-284b-475c-8f28-d8b320be94b3-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9\" (UID: \"bddb43dc-284b-475c-8f28-d8b320be94b3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"
Nov 03 09:01:42.221456 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.220980    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bddb43dc-284b-475c-8f28-d8b320be94b3-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9\" (UID: \"bddb43dc-284b-475c-8f28-d8b320be94b3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"
Nov 03 09:01:42.221456 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.221058    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bddb43dc-284b-475c-8f28-d8b320be94b3-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9\" (UID: \"bddb43dc-284b-475c-8f28-d8b320be94b3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"
Nov 03 09:01:42.254981 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.247004    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"]
Nov 03 09:01:42.322531 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.322503    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bddb43dc-284b-475c-8f28-d8b320be94b3-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9\" (UID: \"bddb43dc-284b-475c-8f28-d8b320be94b3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"
Nov 03 09:01:42.322722 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.322564    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bddb43dc-284b-475c-8f28-d8b320be94b3-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9\" (UID: \"bddb43dc-284b-475c-8f28-d8b320be94b3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"
Nov 03 09:01:42.322722 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.322607    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkrtg\" (UniqueName: \"kubernetes.io/projected/bddb43dc-284b-475c-8f28-d8b320be94b3-kube-api-access-xkrtg\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9\" (UID: \"bddb43dc-284b-475c-8f28-d8b320be94b3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"
Nov 03 09:01:42.322722 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.322644    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bddb43dc-284b-475c-8f28-d8b320be94b3-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9\" (UID: \"bddb43dc-284b-475c-8f28-d8b320be94b3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"
Nov 03 09:01:42.322722 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.322664    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bddb43dc-284b-475c-8f28-d8b320be94b3-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9\" (UID: \"bddb43dc-284b-475c-8f28-d8b320be94b3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"
Nov 03 09:01:42.322722 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.322687    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bddb43dc-284b-475c-8f28-d8b320be94b3-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9\" (UID: \"bddb43dc-284b-475c-8f28-d8b320be94b3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"
Nov 03 09:01:42.322722 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.322714    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bddb43dc-284b-475c-8f28-d8b320be94b3-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9\" (UID: \"bddb43dc-284b-475c-8f28-d8b320be94b3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"
Nov 03 09:01:42.323097 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.322770    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bddb43dc-284b-475c-8f28-d8b320be94b3-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9\" (UID: \"bddb43dc-284b-475c-8f28-d8b320be94b3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"
Nov 03 09:01:42.323097 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.322806    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bddb43dc-284b-475c-8f28-d8b320be94b3-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9\" (UID: \"bddb43dc-284b-475c-8f28-d8b320be94b3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"
Nov 03 09:01:42.323209 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.323164    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bddb43dc-284b-475c-8f28-d8b320be94b3-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9\" (UID: \"bddb43dc-284b-475c-8f28-d8b320be94b3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"
Nov 03 09:01:42.323256 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.323204    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bddb43dc-284b-475c-8f28-d8b320be94b3-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9\" (UID: \"bddb43dc-284b-475c-8f28-d8b320be94b3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"
Nov 03 09:01:42.323330 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.323302    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bddb43dc-284b-475c-8f28-d8b320be94b3-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9\" (UID: \"bddb43dc-284b-475c-8f28-d8b320be94b3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"
Nov 03 09:01:42.323468 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.323398    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bddb43dc-284b-475c-8f28-d8b320be94b3-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9\" (UID: \"bddb43dc-284b-475c-8f28-d8b320be94b3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"
Nov 03 09:01:42.323773 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.323750    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bddb43dc-284b-475c-8f28-d8b320be94b3-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9\" (UID: \"bddb43dc-284b-475c-8f28-d8b320be94b3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"
Nov 03 09:01:42.325144 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.325123    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bddb43dc-284b-475c-8f28-d8b320be94b3-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9\" (UID: \"bddb43dc-284b-475c-8f28-d8b320be94b3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"
Nov 03 09:01:42.325459 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.325441    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bddb43dc-284b-475c-8f28-d8b320be94b3-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9\" (UID: \"bddb43dc-284b-475c-8f28-d8b320be94b3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"
Nov 03 09:01:42.381938 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.381905    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bddb43dc-284b-475c-8f28-d8b320be94b3-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9\" (UID: \"bddb43dc-284b-475c-8f28-d8b320be94b3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"
Nov 03 09:01:42.382079 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.381995    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkrtg\" (UniqueName: \"kubernetes.io/projected/bddb43dc-284b-475c-8f28-d8b320be94b3-kube-api-access-xkrtg\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9\" (UID: \"bddb43dc-284b-475c-8f28-d8b320be94b3\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"
Nov 03 09:01:42.418855 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.418829    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"
Nov 03 09:01:42.853227 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.853205    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"]
Nov 03 09:01:42.853838 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:01:42.853812    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbddb43dc_284b_475c_8f28_d8b320be94b3.slice/crio-12da7615a492c957750bf260a0212855b29d4ca581c141a4ea0c6988b8e30cc6 WatchSource:0}: Error finding container 12da7615a492c957750bf260a0212855b29d4ca581c141a4ea0c6988b8e30cc6: Status 404 returned error can't find the container with id 12da7615a492c957750bf260a0212855b29d4ca581c141a4ea0c6988b8e30cc6
Nov 03 09:01:42.856274 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.856237    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:01:42.856366 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.856309    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:01:42.856366 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.856357    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:01:42.905869 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.905846    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c1d945a-lccxp"]
Nov 03 09:01:42.910786 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.910767    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c1d945a-lccxp"
Nov 03 09:01:42.925745 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.925721    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9" event={"ID":"bddb43dc-284b-475c-8f28-d8b320be94b3","Type":"ContainerStarted","Data":"12da7615a492c957750bf260a0212855b29d4ca581c141a4ea0c6988b8e30cc6"}
Nov 03 09:01:42.931217 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.931190    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0980c8ec-9657-45a6-bd54-d78be27d7b32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c1d945a-lccxp\" (UID: \"0980c8ec-9657-45a6-bd54-d78be27d7b32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c1d945a-lccxp"
Nov 03 09:01:42.931357 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.931337    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0980c8ec-9657-45a6-bd54-d78be27d7b32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c1d945a-lccxp\" (UID: \"0980c8ec-9657-45a6-bd54-d78be27d7b32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c1d945a-lccxp"
Nov 03 09:01:42.931594 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.931568    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq9vw\" (UniqueName: \"kubernetes.io/projected/0980c8ec-9657-45a6-bd54-d78be27d7b32-kube-api-access-gq9vw\") pod \"maas-api-revoke-sub-7c1d945a-lccxp\" (UID: \"0980c8ec-9657-45a6-bd54-d78be27d7b32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c1d945a-lccxp"
Nov 03 09:01:42.967698 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.967669    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:01:42.967877 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.967842    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:01:42.968007 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.967905    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:01:42.971528 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.970911    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:01:42.971528 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:42.971180    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c1d945a-lccxp"]
Nov 03 09:01:43.032461 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:43.032423    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq9vw\" (UniqueName: \"kubernetes.io/projected/0980c8ec-9657-45a6-bd54-d78be27d7b32-kube-api-access-gq9vw\") pod \"maas-api-revoke-sub-7c1d945a-lccxp\" (UID: \"0980c8ec-9657-45a6-bd54-d78be27d7b32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c1d945a-lccxp"
Nov 03 09:01:43.032652 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:43.032478    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0980c8ec-9657-45a6-bd54-d78be27d7b32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c1d945a-lccxp\" (UID: \"0980c8ec-9657-45a6-bd54-d78be27d7b32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c1d945a-lccxp"
Nov 03 09:01:43.032652 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:43.032540    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0980c8ec-9657-45a6-bd54-d78be27d7b32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c1d945a-lccxp\" (UID: \"0980c8ec-9657-45a6-bd54-d78be27d7b32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c1d945a-lccxp"
Nov 03 09:01:43.033273 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:43.033247    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0980c8ec-9657-45a6-bd54-d78be27d7b32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c1d945a-lccxp\" (UID: \"0980c8ec-9657-45a6-bd54-d78be27d7b32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c1d945a-lccxp"
Nov 03 09:01:43.035115 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:43.035092    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0980c8ec-9657-45a6-bd54-d78be27d7b32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c1d945a-lccxp\" (UID: \"0980c8ec-9657-45a6-bd54-d78be27d7b32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c1d945a-lccxp"
Nov 03 09:01:43.092587 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:43.092559    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq9vw\" (UniqueName: \"kubernetes.io/projected/0980c8ec-9657-45a6-bd54-d78be27d7b32-kube-api-access-gq9vw\") pod \"maas-api-revoke-sub-7c1d945a-lccxp\" (UID: \"0980c8ec-9657-45a6-bd54-d78be27d7b32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c1d945a-lccxp"
Nov 03 09:01:43.224555 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:43.224527    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c1d945a-lccxp"
Nov 03 09:01:43.605578 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:43.605553    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c1d945a-lccxp"]
Nov 03 09:01:43.605753 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:01:43.605727    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0980c8ec_9657_45a6_bd54_d78be27d7b32.slice/crio-a7739adf91eb3699ce5e3068aae68b37dbfb5a537b747e95867225cbcdfca932 WatchSource:0}: Error finding container a7739adf91eb3699ce5e3068aae68b37dbfb5a537b747e95867225cbcdfca932: Status 404 returned error can't find the container with id a7739adf91eb3699ce5e3068aae68b37dbfb5a537b747e95867225cbcdfca932
Nov 03 09:01:43.949016 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:43.948981    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c1d945a-lccxp" event={"ID":"0980c8ec-9657-45a6-bd54-d78be27d7b32","Type":"ContainerStarted","Data":"a7739adf91eb3699ce5e3068aae68b37dbfb5a537b747e95867225cbcdfca932"}
Nov 03 09:01:43.950685 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:43.950650    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9" event={"ID":"bddb43dc-284b-475c-8f28-d8b320be94b3","Type":"ContainerStarted","Data":"be36647e9a278d5271fc3c412fd6743fa2b19c1e9979776b925d8596b8514eb4"}
Nov 03 09:01:44.428990 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:44.427740    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"
Nov 03 09:01:44.428990 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:44.427905    4473 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 09:01:44.428990 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:44.427975    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9" podUID="bddb43dc-284b-475c-8f28-d8b320be94b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 09:01:44.986972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:44.980012    4473 generic.go:356] "Generic (PLEG): container finished" podID="0980c8ec-9657-45a6-bd54-d78be27d7b32" containerID="f47ae2d0e5cb43a239460b34423bab7a00111f4e8be7a2e02943f4c518c353f3" exitCode=0
Nov 03 09:01:44.986972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:44.981327    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c1d945a-lccxp" event={"ID":"0980c8ec-9657-45a6-bd54-d78be27d7b32","Type":"ContainerDied","Data":"f47ae2d0e5cb43a239460b34423bab7a00111f4e8be7a2e02943f4c518c353f3"}
Nov 03 09:01:45.131262 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:45.121934    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9" podStartSLOduration=3.121915548 podStartE2EDuration="3.121915548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:01:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:01:44.084400148 +0000 UTC m=+1328.353463613" watchObservedRunningTime="2026-11-03 09:01:45.121915548 +0000 UTC m=+1329.390978984"
Nov 03 09:01:45.420524 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:45.420353    4473 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 09:01:45.420797 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:45.420761    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9" podUID="bddb43dc-284b-475c-8f28-d8b320be94b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 09:01:46.420779 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:46.420744    4473 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 09:01:46.421699 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:46.421643    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9" podUID="bddb43dc-284b-475c-8f28-d8b320be94b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 09:01:46.849156 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:46.849050    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh" podUID="00bab831-c667-415d-b6b0-81fff4d57762" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:01:46.863093 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:46.863033    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn" podUID="035b0461-2dfa-42f4-8ea7-2d1911cddca9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:01:46.895416 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:46.895374    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx" podUID="8e84a474-def6-41dc-8516-5e5ccf42ff46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:01:47.011973 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:47.000739    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c1d945a-lccxp"
Nov 03 09:01:47.013118 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:47.013079    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd" podUID="cf4cf2fc-612a-4f5c-a596-543036dde7e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:01:47.078063 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:47.078016    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c1d945a-lccxp" event={"ID":"0980c8ec-9657-45a6-bd54-d78be27d7b32","Type":"ContainerDied","Data":"a7739adf91eb3699ce5e3068aae68b37dbfb5a537b747e95867225cbcdfca932"}
Nov 03 09:01:47.078348 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:47.078307    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7739adf91eb3699ce5e3068aae68b37dbfb5a537b747e95867225cbcdfca932"
Nov 03 09:01:47.078620 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:47.078605    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c1d945a-lccxp"
Nov 03 09:01:47.093084 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:47.093041    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn" podUID="05a8a676-3011-4597-9b71-b70c3387e2b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:01:47.095831 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:47.095804    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0980c8ec-9657-45a6-bd54-d78be27d7b32-kube-api-access-gq9vw\" (UniqueName: \"kubernetes.io/projected/0980c8ec-9657-45a6-bd54-d78be27d7b32-kube-api-access-gq9vw\") pod \"0980c8ec-9657-45a6-bd54-d78be27d7b32\" (UID: \"0980c8ec-9657-45a6-bd54-d78be27d7b32\") "
Nov 03 09:01:47.096085 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:47.096066    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0980c8ec-9657-45a6-bd54-d78be27d7b32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0980c8ec-9657-45a6-bd54-d78be27d7b32-maas-api-openshift-ca\") pod \"0980c8ec-9657-45a6-bd54-d78be27d7b32\" (UID: \"0980c8ec-9657-45a6-bd54-d78be27d7b32\") "
Nov 03 09:01:47.096303 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:47.096284    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0980c8ec-9657-45a6-bd54-d78be27d7b32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0980c8ec-9657-45a6-bd54-d78be27d7b32-maas-api-xks-ca\") pod \"0980c8ec-9657-45a6-bd54-d78be27d7b32\" (UID: \"0980c8ec-9657-45a6-bd54-d78be27d7b32\") "
Nov 03 09:01:47.096673 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:47.096646    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0980c8ec-9657-45a6-bd54-d78be27d7b32-maas-api-openshift-ca" pod "0980c8ec-9657-45a6-bd54-d78be27d7b32" (UID: "0980c8ec-9657-45a6-bd54-d78be27d7b32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:01:47.096997 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:47.096974    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0980c8ec-9657-45a6-bd54-d78be27d7b32-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:01:47.152992 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:47.144111    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0980c8ec-9657-45a6-bd54-d78be27d7b32-maas-api-xks-ca" pod "0980c8ec-9657-45a6-bd54-d78be27d7b32" (UID: "0980c8ec-9657-45a6-bd54-d78be27d7b32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:01:47.152992 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:47.144230    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0980c8ec-9657-45a6-bd54-d78be27d7b32-kube-api-access-gq9vw" pod "0980c8ec-9657-45a6-bd54-d78be27d7b32" (UID: "0980c8ec-9657-45a6-bd54-d78be27d7b32"). InnerVolumeSpecName "kube-api-access-gq9vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:01:47.200802 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:47.200763    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq9vw\" (UniqueName: \"kubernetes.io/projected/0980c8ec-9657-45a6-bd54-d78be27d7b32-kube-api-access-gq9vw\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:01:47.201071 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:47.201052    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0980c8ec-9657-45a6-bd54-d78be27d7b32-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:01:47.205974 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:01:47.202548    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:01:47.425651 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:47.425533    4473 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 09:01:47.433030 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:47.426211    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9" podUID="bddb43dc-284b-475c-8f28-d8b320be94b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 09:01:47.852127 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:47.852011    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh" podUID="00bab831-c667-415d-b6b0-81fff4d57762" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:01:47.853060 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:47.853036    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh"
Nov 03 09:01:47.854505 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:47.854480    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6" podUID="25c366b3-b919-462b-b69a-82ce9dbd7436" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:01:47.884210 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:47.884168    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn" podUID="035b0461-2dfa-42f4-8ea7-2d1911cddca9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:01:47.905791 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:47.905355    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx" podUID="8e84a474-def6-41dc-8516-5e5ccf42ff46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:01:47.943974 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:47.926837    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn"
Nov 03 09:01:47.944421 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:47.944391    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx"
Nov 03 09:01:48.035981 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:48.031041    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd" podUID="cf4cf2fc-612a-4f5c-a596-543036dde7e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:01:48.035981 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:48.031446    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd"
Nov 03 09:01:48.142973 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:48.134713    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn" podUID="05a8a676-3011-4597-9b71-b70c3387e2b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:01:48.197972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:48.143812    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn"
Nov 03 09:01:48.445846 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:48.215034    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm" podUID="60b1d7ef-16c6-4b6a-973b-4a251593eac7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:01:48.471971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:48.422327    4473 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 09:01:48.496976 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:48.472241    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9" podUID="bddb43dc-284b-475c-8f28-d8b320be94b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 09:01:48.921972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:48.902671    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn" podUID="035b0461-2dfa-42f4-8ea7-2d1911cddca9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 09:01:48.921972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:48.903834    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6" podUID="25c366b3-b919-462b-b69a-82ce9dbd7436" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 03 09:01:48.921972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:48.918477    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6"
Nov 03 09:01:48.921972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:48.919931    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx" podUID="8e84a474-def6-41dc-8516-5e5ccf42ff46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:01:48.943236 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:48.936273    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh" podUID="00bab831-c667-415d-b6b0-81fff4d57762" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 03 09:01:49.141010 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:49.120199    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd" podUID="cf4cf2fc-612a-4f5c-a596-543036dde7e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:01:49.185254 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:49.185157    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn" podUID="05a8a676-3011-4597-9b71-b70c3387e2b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:01:49.479794 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:49.479688    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm" podUID="60b1d7ef-16c6-4b6a-973b-4a251593eac7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:01:49.563186 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:49.563136    4473 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 09:01:49.665851 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:49.643460    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm"
Nov 03 09:01:49.688969 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:49.687855    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9" podUID="bddb43dc-284b-475c-8f28-d8b320be94b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 09:01:49.905973 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:49.905027    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6" podUID="25c366b3-b919-462b-b69a-82ce9dbd7436" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 03 09:01:49.906249 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:49.906190    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn" podUID="035b0461-2dfa-42f4-8ea7-2d1911cddca9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 09:01:49.957338 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:49.956293    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx" podUID="8e84a474-def6-41dc-8516-5e5ccf42ff46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:01:49.983971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:49.958054    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh" podUID="00bab831-c667-415d-b6b0-81fff4d57762" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:01:50.007678 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:50.007638    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd"
Nov 03 09:01:50.034970 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:50.007681    4473 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6" podUID="25c366b3-b919-462b-b69a-82ce9dbd7436" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:01:50.098143 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:50.098111    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm"
Nov 03 09:01:50.158024 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:50.153704    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn"
Nov 03 09:01:50.374723 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:50.374698    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx"
Nov 03 09:01:50.409447 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:50.409364    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh"
Nov 03 09:01:50.411510 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:50.411479    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6"
Nov 03 09:01:50.446154 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:50.446127    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn"
Nov 03 09:01:50.454336 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:50.454307    4473 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 09:01:50.454597 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:50.454568    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9" podUID="bddb43dc-284b-475c-8f28-d8b320be94b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 09:01:51.420024 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:51.419987    4473 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Nov 03 09:01:51.420532 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:51.420052    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9" podUID="bddb43dc-284b-475c-8f28-d8b320be94b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Nov 03 09:01:53.380516 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:53.380483    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"
Nov 03 09:01:53.380963 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:53.380790    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"
Nov 03 09:01:53.382650 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:53.382627    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"
Nov 03 09:01:53.672201 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:53.672106    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0803d9e-d255b"]
Nov 03 09:01:53.672870 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:53.672841    4473 cpu_manager.go:414] "containerMap: removing container" podUID="0980c8ec-9657-45a6-bd54-d78be27d7b32" containerName="revoke-keys"
Nov 03 09:01:53.672870 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:53.672864    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0980c8ec-9657-45a6-bd54-d78be27d7b32" containerName="revoke-keys"
Nov 03 09:01:53.673154 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:53.673042    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="0980c8ec-9657-45a6-bd54-d78be27d7b32" containerName="revoke-keys"
Nov 03 09:01:53.677624 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:53.677604    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0803d9e-d255b"
Nov 03 09:01:53.697350 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:53.697319    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpl72\" (UniqueName: \"kubernetes.io/projected/66c905c8-ee01-442a-ab3d-27167deb93ce-kube-api-access-kpl72\") pod \"maas-api-revoke-sub-a0803d9e-d255b\" (UID: \"66c905c8-ee01-442a-ab3d-27167deb93ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0803d9e-d255b"
Nov 03 09:01:53.697483 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:53.697362    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66c905c8-ee01-442a-ab3d-27167deb93ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0803d9e-d255b\" (UID: \"66c905c8-ee01-442a-ab3d-27167deb93ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0803d9e-d255b"
Nov 03 09:01:53.697648 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:53.697622    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66c905c8-ee01-442a-ab3d-27167deb93ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0803d9e-d255b\" (UID: \"66c905c8-ee01-442a-ab3d-27167deb93ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0803d9e-d255b"
Nov 03 09:01:53.732549 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:53.732518    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:01:53.732731 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:53.732585    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:01:53.733011 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:53.732987    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:01:53.733413 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:53.733079    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:01:53.734359 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:53.734337    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0803d9e-d255b"]
Nov 03 09:01:53.799322 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:53.799292    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpl72\" (UniqueName: \"kubernetes.io/projected/66c905c8-ee01-442a-ab3d-27167deb93ce-kube-api-access-kpl72\") pod \"maas-api-revoke-sub-a0803d9e-d255b\" (UID: \"66c905c8-ee01-442a-ab3d-27167deb93ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0803d9e-d255b"
Nov 03 09:01:53.799472 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:53.799327    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66c905c8-ee01-442a-ab3d-27167deb93ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0803d9e-d255b\" (UID: \"66c905c8-ee01-442a-ab3d-27167deb93ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0803d9e-d255b"
Nov 03 09:01:53.799472 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:53.799461    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66c905c8-ee01-442a-ab3d-27167deb93ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0803d9e-d255b\" (UID: \"66c905c8-ee01-442a-ab3d-27167deb93ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0803d9e-d255b"
Nov 03 09:01:53.800045 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:53.800026    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66c905c8-ee01-442a-ab3d-27167deb93ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0803d9e-d255b\" (UID: \"66c905c8-ee01-442a-ab3d-27167deb93ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0803d9e-d255b"
Nov 03 09:01:53.801937 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:53.801919    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66c905c8-ee01-442a-ab3d-27167deb93ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0803d9e-d255b\" (UID: \"66c905c8-ee01-442a-ab3d-27167deb93ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0803d9e-d255b"
Nov 03 09:01:53.859132 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:53.859093    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpl72\" (UniqueName: \"kubernetes.io/projected/66c905c8-ee01-442a-ab3d-27167deb93ce-kube-api-access-kpl72\") pod \"maas-api-revoke-sub-a0803d9e-d255b\" (UID: \"66c905c8-ee01-442a-ab3d-27167deb93ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0803d9e-d255b"
Nov 03 09:01:53.998663 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:53.998587    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0803d9e-d255b"
Nov 03 09:01:54.376687 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:54.376666    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0803d9e-d255b"]
Nov 03 09:01:54.377705 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:01:54.377680    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c905c8_ee01_442a_ab3d_27167deb93ce.slice/crio-0c2b968fd584ed33287c11f77bfc0ba2d9de66675e77dfe41ff7084f1e09a8f6 WatchSource:0}: Error finding container 0c2b968fd584ed33287c11f77bfc0ba2d9de66675e77dfe41ff7084f1e09a8f6: Status 404 returned error can't find the container with id 0c2b968fd584ed33287c11f77bfc0ba2d9de66675e77dfe41ff7084f1e09a8f6
Nov 03 09:01:54.424739 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:54.424712    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0803d9e-d255b" event={"ID":"66c905c8-ee01-442a-ab3d-27167deb93ce","Type":"ContainerStarted","Data":"0c2b968fd584ed33287c11f77bfc0ba2d9de66675e77dfe41ff7084f1e09a8f6"}
Nov 03 09:01:55.432483 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:55.432440    4473 generic.go:356] "Generic (PLEG): container finished" podID="66c905c8-ee01-442a-ab3d-27167deb93ce" containerID="2d6700d493748cb2ca4bfaae2713b57bc81f17dd0d2ec4b90656312e9c3c7381" exitCode=0
Nov 03 09:01:55.432863 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:55.432487    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0803d9e-d255b" event={"ID":"66c905c8-ee01-442a-ab3d-27167deb93ce","Type":"ContainerDied","Data":"2d6700d493748cb2ca4bfaae2713b57bc81f17dd0d2ec4b90656312e9c3c7381"}
Nov 03 09:01:56.586119 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:56.586098    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0803d9e-d255b"
Nov 03 09:01:56.647891 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:56.647852    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66c905c8-ee01-442a-ab3d-27167deb93ce-kube-api-access-kpl72\" (UniqueName: \"kubernetes.io/projected/66c905c8-ee01-442a-ab3d-27167deb93ce-kube-api-access-kpl72\") pod \"66c905c8-ee01-442a-ab3d-27167deb93ce\" (UID: \"66c905c8-ee01-442a-ab3d-27167deb93ce\") "
Nov 03 09:01:56.648043 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:56.647933    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66c905c8-ee01-442a-ab3d-27167deb93ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66c905c8-ee01-442a-ab3d-27167deb93ce-maas-api-xks-ca\") pod \"66c905c8-ee01-442a-ab3d-27167deb93ce\" (UID: \"66c905c8-ee01-442a-ab3d-27167deb93ce\") "
Nov 03 09:01:56.648043 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:56.648036    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66c905c8-ee01-442a-ab3d-27167deb93ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66c905c8-ee01-442a-ab3d-27167deb93ce-maas-api-openshift-ca\") pod \"66c905c8-ee01-442a-ab3d-27167deb93ce\" (UID: \"66c905c8-ee01-442a-ab3d-27167deb93ce\") "
Nov 03 09:01:56.648385 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:56.648361    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66c905c8-ee01-442a-ab3d-27167deb93ce-maas-api-openshift-ca" pod "66c905c8-ee01-442a-ab3d-27167deb93ce" (UID: "66c905c8-ee01-442a-ab3d-27167deb93ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:01:56.650136 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:56.650114    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66c905c8-ee01-442a-ab3d-27167deb93ce-maas-api-xks-ca" pod "66c905c8-ee01-442a-ab3d-27167deb93ce" (UID: "66c905c8-ee01-442a-ab3d-27167deb93ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:01:56.650202 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:56.650162    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66c905c8-ee01-442a-ab3d-27167deb93ce-kube-api-access-kpl72" pod "66c905c8-ee01-442a-ab3d-27167deb93ce" (UID: "66c905c8-ee01-442a-ab3d-27167deb93ce"). InnerVolumeSpecName "kube-api-access-kpl72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:01:56.749553 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:56.749480    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66c905c8-ee01-442a-ab3d-27167deb93ce-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:01:56.749553 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:56.749506    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpl72\" (UniqueName: \"kubernetes.io/projected/66c905c8-ee01-442a-ab3d-27167deb93ce-kube-api-access-kpl72\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:01:56.749553 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:56.749516    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66c905c8-ee01-442a-ab3d-27167deb93ce-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:01:57.447010 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:57.446981    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0803d9e-d255b"
Nov 03 09:01:57.447010 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:57.446989    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0803d9e-d255b" event={"ID":"66c905c8-ee01-442a-ab3d-27167deb93ce","Type":"ContainerDied","Data":"0c2b968fd584ed33287c11f77bfc0ba2d9de66675e77dfe41ff7084f1e09a8f6"}
Nov 03 09:01:57.447188 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:01:57.447044    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c2b968fd584ed33287c11f77bfc0ba2d9de66675e77dfe41ff7084f1e09a8f6"
Nov 03 09:02:01.615328 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:01.615298    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"]
Nov 03 09:02:01.615914 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:01.615521    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr" podUID="165b0188-6916-4776-9059-7a4f765a13f0" containerName="istio-proxy" containerID="cri-o://ceac0af690d74ef235767727b311264e38c0c40bdb70a4b3217f0427856d585b" gracePeriod=30
Nov 03 09:02:05.408217 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:05.408184    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:02:06.813529 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:06.813132    4473 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Nov 03 09:02:06.813529 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:06.813213    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr" podUID="165b0188-6916-4776-9059-7a4f765a13f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Nov 03 09:02:07.722986 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:07.722732    4473 generic.go:356] "Generic (PLEG): container finished" podID="165b0188-6916-4776-9059-7a4f765a13f0" containerID="ceac0af690d74ef235767727b311264e38c0c40bdb70a4b3217f0427856d585b" exitCode=0
Nov 03 09:02:07.722986 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:07.722896    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr" event={"ID":"165b0188-6916-4776-9059-7a4f765a13f0","Type":"ContainerDied","Data":"ceac0af690d74ef235767727b311264e38c0c40bdb70a4b3217f0427856d585b"}
Nov 03 09:02:07.905449 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:07.905417    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"
Nov 03 09:02:07.906347 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:07.906311    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn" podUID="035b0461-2dfa-42f4-8ea7-2d1911cddca9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:02:07.928280 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:07.928244    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx" podUID="8e84a474-def6-41dc-8516-5e5ccf42ff46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 09:02:07.942761 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:07.942719    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/165b0188-6916-4776-9059-7a4f765a13f0-istio-token\" (UniqueName: \"kubernetes.io/projected/165b0188-6916-4776-9059-7a4f765a13f0-istio-token\") pod \"165b0188-6916-4776-9059-7a4f765a13f0\" (UID: \"165b0188-6916-4776-9059-7a4f765a13f0\") "
Nov 03 09:02:07.943468 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:07.943447    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/165b0188-6916-4776-9059-7a4f765a13f0-kube-api-access-6jbll\" (UniqueName: \"kubernetes.io/projected/165b0188-6916-4776-9059-7a4f765a13f0-kube-api-access-6jbll\") pod \"165b0188-6916-4776-9059-7a4f765a13f0\" (UID: \"165b0188-6916-4776-9059-7a4f765a13f0\") "
Nov 03 09:02:07.944022 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:07.944003    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/165b0188-6916-4776-9059-7a4f765a13f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/165b0188-6916-4776-9059-7a4f765a13f0-credential-socket\") pod \"165b0188-6916-4776-9059-7a4f765a13f0\" (UID: \"165b0188-6916-4776-9059-7a4f765a13f0\") "
Nov 03 09:02:07.944484 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:07.944370    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165b0188-6916-4776-9059-7a4f765a13f0-credential-socket" pod "165b0188-6916-4776-9059-7a4f765a13f0" (UID: "165b0188-6916-4776-9059-7a4f765a13f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:02:07.944658 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:07.944641    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/165b0188-6916-4776-9059-7a4f765a13f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/165b0188-6916-4776-9059-7a4f765a13f0-istio-podinfo\") pod \"165b0188-6916-4776-9059-7a4f765a13f0\" (UID: \"165b0188-6916-4776-9059-7a4f765a13f0\") "
Nov 03 09:02:07.945148 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:07.945130    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/165b0188-6916-4776-9059-7a4f765a13f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/165b0188-6916-4776-9059-7a4f765a13f0-istio-data\") pod \"165b0188-6916-4776-9059-7a4f765a13f0\" (UID: \"165b0188-6916-4776-9059-7a4f765a13f0\") "
Nov 03 09:02:07.960653 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:07.960632    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/165b0188-6916-4776-9059-7a4f765a13f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/165b0188-6916-4776-9059-7a4f765a13f0-istiod-ca-cert\") pod \"165b0188-6916-4776-9059-7a4f765a13f0\" (UID: \"165b0188-6916-4776-9059-7a4f765a13f0\") "
Nov 03 09:02:07.960887 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:07.960865    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/165b0188-6916-4776-9059-7a4f765a13f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/165b0188-6916-4776-9059-7a4f765a13f0-workload-socket\") pod \"165b0188-6916-4776-9059-7a4f765a13f0\" (UID: \"165b0188-6916-4776-9059-7a4f765a13f0\") "
Nov 03 09:02:07.961113 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:07.961093    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/165b0188-6916-4776-9059-7a4f765a13f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/165b0188-6916-4776-9059-7a4f765a13f0-istio-envoy\") pod \"165b0188-6916-4776-9059-7a4f765a13f0\" (UID: \"165b0188-6916-4776-9059-7a4f765a13f0\") "
Nov 03 09:02:07.961265 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:07.961249    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/165b0188-6916-4776-9059-7a4f765a13f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/165b0188-6916-4776-9059-7a4f765a13f0-workload-certs\") pod \"165b0188-6916-4776-9059-7a4f765a13f0\" (UID: \"165b0188-6916-4776-9059-7a4f765a13f0\") "
Nov 03 09:02:07.961897 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:07.961873    4473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/165b0188-6916-4776-9059-7a4f765a13f0-credential-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:02:07.964069 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:07.951552    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh" podUID="00bab831-c667-415d-b6b0-81fff4d57762" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:02:07.964637 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:07.960268    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd" podUID="cf4cf2fc-612a-4f5c-a596-543036dde7e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:02:07.964637 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:07.960361    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165b0188-6916-4776-9059-7a4f765a13f0-istio-data" pod "165b0188-6916-4776-9059-7a4f765a13f0" (UID: "165b0188-6916-4776-9059-7a4f765a13f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:02:07.964637 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:07.963184    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165b0188-6916-4776-9059-7a4f765a13f0-workload-socket" pod "165b0188-6916-4776-9059-7a4f765a13f0" (UID: "165b0188-6916-4776-9059-7a4f765a13f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:02:07.964637 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:07.963425    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/165b0188-6916-4776-9059-7a4f765a13f0-istiod-ca-cert" pod "165b0188-6916-4776-9059-7a4f765a13f0" (UID: "165b0188-6916-4776-9059-7a4f765a13f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:02:07.964637 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:07.964606    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165b0188-6916-4776-9059-7a4f765a13f0-workload-certs" pod "165b0188-6916-4776-9059-7a4f765a13f0" (UID: "165b0188-6916-4776-9059-7a4f765a13f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:02:07.986203 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:07.980425    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/165b0188-6916-4776-9059-7a4f765a13f0-kube-api-access-6jbll" pod "165b0188-6916-4776-9059-7a4f765a13f0" (UID: "165b0188-6916-4776-9059-7a4f765a13f0"). InnerVolumeSpecName "kube-api-access-6jbll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:02:07.989546 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:07.989468    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/165b0188-6916-4776-9059-7a4f765a13f0-istio-podinfo" pod "165b0188-6916-4776-9059-7a4f765a13f0" (UID: "165b0188-6916-4776-9059-7a4f765a13f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 09:02:08.009986 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:08.002526    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/165b0188-6916-4776-9059-7a4f765a13f0-istio-token" pod "165b0188-6916-4776-9059-7a4f765a13f0" (UID: "165b0188-6916-4776-9059-7a4f765a13f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:02:08.009986 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:08.002968    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165b0188-6916-4776-9059-7a4f765a13f0-istio-envoy" pod "165b0188-6916-4776-9059-7a4f765a13f0" (UID: "165b0188-6916-4776-9059-7a4f765a13f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:02:08.079972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:08.079731    4473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/165b0188-6916-4776-9059-7a4f765a13f0-istio-data\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:02:08.079972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:08.079770    4473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/165b0188-6916-4776-9059-7a4f765a13f0-istiod-ca-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:02:08.079972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:08.079785    4473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/165b0188-6916-4776-9059-7a4f765a13f0-workload-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:02:08.079972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:08.079799    4473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/165b0188-6916-4776-9059-7a4f765a13f0-istio-envoy\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:02:08.079972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:08.079813    4473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/165b0188-6916-4776-9059-7a4f765a13f0-workload-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:02:08.079972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:08.079825    4473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/165b0188-6916-4776-9059-7a4f765a13f0-istio-token\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:02:08.079972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:08.079839    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jbll\" (UniqueName: \"kubernetes.io/projected/165b0188-6916-4776-9059-7a4f765a13f0-kube-api-access-6jbll\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:02:08.079972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:08.079854    4473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/165b0188-6916-4776-9059-7a4f765a13f0-istio-podinfo\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:02:08.121970 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:08.112930    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm" podUID="60b1d7ef-16c6-4b6a-973b-4a251593eac7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:02:08.159889 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:08.159078    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn" podUID="05a8a676-3011-4597-9b71-b70c3387e2b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 03 09:02:08.836415 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:08.836378    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr" event={"ID":"165b0188-6916-4776-9059-7a4f765a13f0","Type":"ContainerDied","Data":"ae0ec75c5c110df1fd10f8369e53e6dd281f057d14cf8857814f9ba81d21f5c3"}
Nov 03 09:02:08.836789 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:08.836759    4473 scope.go:122] "RemoveContainer" containerID="ceac0af690d74ef235767727b311264e38c0c40bdb70a4b3217f0427856d585b"
Nov 03 09:02:08.837239 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:08.837219    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"
Nov 03 09:02:09.532972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:09.517095    4473 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 09:02:09.532972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:09.517181    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9" podUID="bddb43dc-284b-475c-8f28-d8b320be94b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:02:09.613980 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:09.610336    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"]
Nov 03 09:02:09.741766 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:09.741670    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-49mpr"]
Nov 03 09:02:09.894971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:09.890192    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="165b0188-6916-4776-9059-7a4f765a13f0" path="/var/lib/kubelet/pods/165b0188-6916-4776-9059-7a4f765a13f0/volumes"
Nov 03 09:02:09.928073 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:09.928026    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx" podUID="8e84a474-def6-41dc-8516-5e5ccf42ff46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 09:02:09.949088 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:09.949037    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh" podUID="00bab831-c667-415d-b6b0-81fff4d57762" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:02:10.097157 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:10.097044    4473 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6" podUID="25c366b3-b919-462b-b69a-82ce9dbd7436" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:02:10.101417 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:10.100724    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm" podUID="60b1d7ef-16c6-4b6a-973b-4a251593eac7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:02:11.990285 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:11.990256    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d1ebae5-rvxks"]
Nov 03 09:02:11.990741 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:11.990727    4473 cpu_manager.go:414] "containerMap: removing container" podUID="66c905c8-ee01-442a-ab3d-27167deb93ce" containerName="revoke-keys"
Nov 03 09:02:11.990784 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:11.990742    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66c905c8-ee01-442a-ab3d-27167deb93ce" containerName="revoke-keys"
Nov 03 09:02:11.990784 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:11.990762    4473 cpu_manager.go:414] "containerMap: removing container" podUID="165b0188-6916-4776-9059-7a4f765a13f0" containerName="istio-proxy"
Nov 03 09:02:11.990784 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:11.990767    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="165b0188-6916-4776-9059-7a4f765a13f0" containerName="istio-proxy"
Nov 03 09:02:11.990883 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:11.990836    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="66c905c8-ee01-442a-ab3d-27167deb93ce" containerName="revoke-keys"
Nov 03 09:02:11.990883 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:11.990846    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="165b0188-6916-4776-9059-7a4f765a13f0" containerName="istio-proxy"
Nov 03 09:02:11.995224 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:11.995207    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1ebae5-rvxks"
Nov 03 09:02:12.029006 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.028937    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp9bv\" (UniqueName: \"kubernetes.io/projected/dd482794-363d-4374-a3a5-e0848799005e-kube-api-access-fp9bv\") pod \"maas-api-revoke-sub-9d1ebae5-rvxks\" (UID: \"dd482794-363d-4374-a3a5-e0848799005e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1ebae5-rvxks"
Nov 03 09:02:12.029177 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.029071    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd482794-363d-4374-a3a5-e0848799005e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d1ebae5-rvxks\" (UID: \"dd482794-363d-4374-a3a5-e0848799005e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1ebae5-rvxks"
Nov 03 09:02:12.029276 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.029259    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd482794-363d-4374-a3a5-e0848799005e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d1ebae5-rvxks\" (UID: \"dd482794-363d-4374-a3a5-e0848799005e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1ebae5-rvxks"
Nov 03 09:02:12.053502 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.053474    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d1ebae5-rvxks"]
Nov 03 09:02:12.099979 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.099936    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:02:12.100217 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.099963    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:02:12.100217 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.099937    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:02:12.100400 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.099977    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:02:12.130841 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.130811    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp9bv\" (UniqueName: \"kubernetes.io/projected/dd482794-363d-4374-a3a5-e0848799005e-kube-api-access-fp9bv\") pod \"maas-api-revoke-sub-9d1ebae5-rvxks\" (UID: \"dd482794-363d-4374-a3a5-e0848799005e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1ebae5-rvxks"
Nov 03 09:02:12.131014 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.130860    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd482794-363d-4374-a3a5-e0848799005e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d1ebae5-rvxks\" (UID: \"dd482794-363d-4374-a3a5-e0848799005e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1ebae5-rvxks"
Nov 03 09:02:12.131014 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.130986    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd482794-363d-4374-a3a5-e0848799005e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d1ebae5-rvxks\" (UID: \"dd482794-363d-4374-a3a5-e0848799005e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1ebae5-rvxks"
Nov 03 09:02:12.131537 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.131515    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd482794-363d-4374-a3a5-e0848799005e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d1ebae5-rvxks\" (UID: \"dd482794-363d-4374-a3a5-e0848799005e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1ebae5-rvxks"
Nov 03 09:02:12.133481 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.133456    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd482794-363d-4374-a3a5-e0848799005e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d1ebae5-rvxks\" (UID: \"dd482794-363d-4374-a3a5-e0848799005e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1ebae5-rvxks"
Nov 03 09:02:12.190252 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.190223    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp9bv\" (UniqueName: \"kubernetes.io/projected/dd482794-363d-4374-a3a5-e0848799005e-kube-api-access-fp9bv\") pod \"maas-api-revoke-sub-9d1ebae5-rvxks\" (UID: \"dd482794-363d-4374-a3a5-e0848799005e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1ebae5-rvxks"
Nov 03 09:02:12.308810 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.308733    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1ebae5-rvxks"
Nov 03 09:02:12.356332 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.356295    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"]
Nov 03 09:02:12.360347 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.360323    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"
Nov 03 09:02:12.433924 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.433888    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6\" (UID: \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"
Nov 03 09:02:12.434147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.433934    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6\" (UID: \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"
Nov 03 09:02:12.434147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.434053    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6\" (UID: \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"
Nov 03 09:02:12.434147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.434128    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6\" (UID: \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"
Nov 03 09:02:12.434271 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.434228    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr9x9\" (UniqueName: \"kubernetes.io/projected/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-kube-api-access-sr9x9\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6\" (UID: \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"
Nov 03 09:02:12.434315 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.434295    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6\" (UID: \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"
Nov 03 09:02:12.434381 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.434367    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6\" (UID: \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"
Nov 03 09:02:12.434425 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.434393    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6\" (UID: \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"
Nov 03 09:02:12.434425 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.434412    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6\" (UID: \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"
Nov 03 09:02:12.462501 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.462453    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"]
Nov 03 09:02:12.535140 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.535096    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6\" (UID: \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"
Nov 03 09:02:12.535336 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.535153    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6\" (UID: \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"
Nov 03 09:02:12.535336 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.535323    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6\" (UID: \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"
Nov 03 09:02:12.535454 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.535412    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6\" (UID: \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"
Nov 03 09:02:12.535531 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.535513    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr9x9\" (UniqueName: \"kubernetes.io/projected/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-kube-api-access-sr9x9\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6\" (UID: \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"
Nov 03 09:02:12.535634 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.535617    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6\" (UID: \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"
Nov 03 09:02:12.535703 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.535629    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6\" (UID: \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"
Nov 03 09:02:12.535703 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.535681    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6\" (UID: \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"
Nov 03 09:02:12.535814 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.535708    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6\" (UID: \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"
Nov 03 09:02:12.535814 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.535735    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6\" (UID: \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"
Nov 03 09:02:12.536037 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.536013    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6\" (UID: \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"
Nov 03 09:02:12.536145 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.536107    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6\" (UID: \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"
Nov 03 09:02:12.536145 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.536134    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6\" (UID: \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"
Nov 03 09:02:12.536217 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.536139    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6\" (UID: \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"
Nov 03 09:02:12.537846 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.537831    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6\" (UID: \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"
Nov 03 09:02:12.537912 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.537895    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6\" (UID: \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"
Nov 03 09:02:12.595330 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.595256    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr9x9\" (UniqueName: \"kubernetes.io/projected/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-kube-api-access-sr9x9\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6\" (UID: \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"
Nov 03 09:02:12.595454 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.595374    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6\" (UID: \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"
Nov 03 09:02:12.685397 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.685372    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d1ebae5-rvxks"]
Nov 03 09:02:12.685622 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:02:12.685598    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd482794_363d_4374_a3a5_e0848799005e.slice/crio-b057581bebe924a601c30d0f5be03ff1da7e6254921e8cc5dab951812fbc9884 WatchSource:0}: Error finding container b057581bebe924a601c30d0f5be03ff1da7e6254921e8cc5dab951812fbc9884: Status 404 returned error can't find the container with id b057581bebe924a601c30d0f5be03ff1da7e6254921e8cc5dab951812fbc9884
Nov 03 09:02:12.689523 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.689502    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"
Nov 03 09:02:12.883997 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:12.883958    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1ebae5-rvxks" event={"ID":"dd482794-363d-4374-a3a5-e0848799005e","Type":"ContainerStarted","Data":"b057581bebe924a601c30d0f5be03ff1da7e6254921e8cc5dab951812fbc9884"}
Nov 03 09:02:13.124456 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:13.124399    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"]
Nov 03 09:02:13.124805 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:02:13.124507    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f53106b_dc09_41eb_ab1a_16cd3fa8de90.slice/crio-8eeec49b1686c744bd4541d304c64c359cf26768753928a3cf07b714b54ced88 WatchSource:0}: Error finding container 8eeec49b1686c744bd4541d304c64c359cf26768753928a3cf07b714b54ced88: Status 404 returned error can't find the container with id 8eeec49b1686c744bd4541d304c64c359cf26768753928a3cf07b714b54ced88
Nov 03 09:02:13.127105 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:13.127078    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:02:13.127174 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:13.127144    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:02:13.127210 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:13.127189    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:02:13.891555 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:13.891521    4473 generic.go:356] "Generic (PLEG): container finished" podID="dd482794-363d-4374-a3a5-e0848799005e" containerID="34cc879024d822cd92de15349a0336bc25df1b4730ff66d36b44690d018ec013" exitCode=0
Nov 03 09:02:13.891741 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:13.891616    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1ebae5-rvxks" event={"ID":"dd482794-363d-4374-a3a5-e0848799005e","Type":"ContainerDied","Data":"34cc879024d822cd92de15349a0336bc25df1b4730ff66d36b44690d018ec013"}
Nov 03 09:02:13.896644 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:13.896617    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6" event={"ID":"5f53106b-dc09-41eb-ab1a-16cd3fa8de90","Type":"ContainerStarted","Data":"d5ca5ebee09815d7292df4bb424dc78fefd86541767e96f9166851329f93f736"}
Nov 03 09:02:13.896815 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:13.896662    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6" event={"ID":"5f53106b-dc09-41eb-ab1a-16cd3fa8de90","Type":"ContainerStarted","Data":"8eeec49b1686c744bd4541d304c64c359cf26768753928a3cf07b714b54ced88"}
Nov 03 09:02:14.144544 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:14.144489    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6" podStartSLOduration=2.144474377 podStartE2EDuration="2.144474377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:02:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:02:14.141544411 +0000 UTC m=+1358.410607873" watchObservedRunningTime="2026-11-03 09:02:14.144474377 +0000 UTC m=+1358.413537902"
Nov 03 09:02:14.690504 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:14.690466    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"
Nov 03 09:02:14.692249 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:14.692223    4473 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 09:02:14.692397 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:14.692284    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6" podUID="5f53106b-dc09-41eb-ab1a-16cd3fa8de90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 09:02:15.087554 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:15.087533    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1ebae5-rvxks"
Nov 03 09:02:15.169443 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:15.169407    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd482794-363d-4374-a3a5-e0848799005e-kube-api-access-fp9bv\" (UniqueName: \"kubernetes.io/projected/dd482794-363d-4374-a3a5-e0848799005e-kube-api-access-fp9bv\") pod \"dd482794-363d-4374-a3a5-e0848799005e\" (UID: \"dd482794-363d-4374-a3a5-e0848799005e\") "
Nov 03 09:02:15.169890 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:15.169474    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd482794-363d-4374-a3a5-e0848799005e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd482794-363d-4374-a3a5-e0848799005e-maas-api-openshift-ca\") pod \"dd482794-363d-4374-a3a5-e0848799005e\" (UID: \"dd482794-363d-4374-a3a5-e0848799005e\") "
Nov 03 09:02:15.169890 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:15.169591    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd482794-363d-4374-a3a5-e0848799005e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd482794-363d-4374-a3a5-e0848799005e-maas-api-xks-ca\") pod \"dd482794-363d-4374-a3a5-e0848799005e\" (UID: \"dd482794-363d-4374-a3a5-e0848799005e\") "
Nov 03 09:02:15.169890 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:15.169729    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd482794-363d-4374-a3a5-e0848799005e-maas-api-openshift-ca" pod "dd482794-363d-4374-a3a5-e0848799005e" (UID: "dd482794-363d-4374-a3a5-e0848799005e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:02:15.169890 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:15.169845    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd482794-363d-4374-a3a5-e0848799005e-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:02:15.171774 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:15.171750    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd482794-363d-4374-a3a5-e0848799005e-kube-api-access-fp9bv" pod "dd482794-363d-4374-a3a5-e0848799005e" (UID: "dd482794-363d-4374-a3a5-e0848799005e"). InnerVolumeSpecName "kube-api-access-fp9bv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:02:15.171876 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:15.171826    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd482794-363d-4374-a3a5-e0848799005e-maas-api-xks-ca" pod "dd482794-363d-4374-a3a5-e0848799005e" (UID: "dd482794-363d-4374-a3a5-e0848799005e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:02:15.271058 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:15.270979    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp9bv\" (UniqueName: \"kubernetes.io/projected/dd482794-363d-4374-a3a5-e0848799005e-kube-api-access-fp9bv\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:02:15.271231 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:15.271218    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd482794-363d-4374-a3a5-e0848799005e-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:02:15.690506 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:15.690470    4473 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 09:02:15.690681 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:15.690524    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6" podUID="5f53106b-dc09-41eb-ab1a-16cd3fa8de90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 09:02:15.910748 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:15.910724    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1ebae5-rvxks"
Nov 03 09:02:15.910914 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:15.910842    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1ebae5-rvxks" event={"ID":"dd482794-363d-4374-a3a5-e0848799005e","Type":"ContainerDied","Data":"b057581bebe924a601c30d0f5be03ff1da7e6254921e8cc5dab951812fbc9884"}
Nov 03 09:02:15.910914 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:15.910881    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b057581bebe924a601c30d0f5be03ff1da7e6254921e8cc5dab951812fbc9884"
Nov 03 09:02:16.689873 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:16.689842    4473 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 09:02:16.690278 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:16.689892    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6" podUID="5f53106b-dc09-41eb-ab1a-16cd3fa8de90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 09:02:16.964726 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:02:16.964642    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:02:17.105446 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.105411    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"]
Nov 03 09:02:17.105902 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.105890    4473 cpu_manager.go:414] "containerMap: removing container" podUID="dd482794-363d-4374-a3a5-e0848799005e" containerName="revoke-keys"
Nov 03 09:02:17.105961 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.105904    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd482794-363d-4374-a3a5-e0848799005e" containerName="revoke-keys"
Nov 03 09:02:17.106002 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.105991    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd482794-363d-4374-a3a5-e0848799005e" containerName="revoke-keys"
Nov 03 09:02:17.113998 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.112179    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"
Nov 03 09:02:17.171046 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.171012    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"]
Nov 03 09:02:17.195216 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.195188    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ecc8182-a9fd-471f-a00d-2195f2757dfd-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8\" (UID: \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"
Nov 03 09:02:17.195449 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.195436    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ecc8182-a9fd-471f-a00d-2195f2757dfd-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8\" (UID: \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"
Nov 03 09:02:17.195557 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.195546    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ecc8182-a9fd-471f-a00d-2195f2757dfd-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8\" (UID: \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"
Nov 03 09:02:17.195704 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.195674    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ecc8182-a9fd-471f-a00d-2195f2757dfd-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8\" (UID: \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"
Nov 03 09:02:17.195970 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.195936    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ecc8182-a9fd-471f-a00d-2195f2757dfd-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8\" (UID: \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"
Nov 03 09:02:17.196071 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.196056    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfj87\" (UniqueName: \"kubernetes.io/projected/4ecc8182-a9fd-471f-a00d-2195f2757dfd-kube-api-access-sfj87\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8\" (UID: \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"
Nov 03 09:02:17.196134 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.196113    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ecc8182-a9fd-471f-a00d-2195f2757dfd-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8\" (UID: \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"
Nov 03 09:02:17.196176 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.196153    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ecc8182-a9fd-471f-a00d-2195f2757dfd-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8\" (UID: \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"
Nov 03 09:02:17.196219 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.196207    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ecc8182-a9fd-471f-a00d-2195f2757dfd-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8\" (UID: \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"
Nov 03 09:02:17.297728 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.297638    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ecc8182-a9fd-471f-a00d-2195f2757dfd-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8\" (UID: \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"
Nov 03 09:02:17.297728 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.297685    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ecc8182-a9fd-471f-a00d-2195f2757dfd-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8\" (UID: \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"
Nov 03 09:02:17.297728 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.297707    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ecc8182-a9fd-471f-a00d-2195f2757dfd-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8\" (UID: \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"
Nov 03 09:02:17.298046 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.297780    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ecc8182-a9fd-471f-a00d-2195f2757dfd-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8\" (UID: \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"
Nov 03 09:02:17.298046 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.297806    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfj87\" (UniqueName: \"kubernetes.io/projected/4ecc8182-a9fd-471f-a00d-2195f2757dfd-kube-api-access-sfj87\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8\" (UID: \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"
Nov 03 09:02:17.298046 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.297851    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ecc8182-a9fd-471f-a00d-2195f2757dfd-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8\" (UID: \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"
Nov 03 09:02:17.298046 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.297880    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ecc8182-a9fd-471f-a00d-2195f2757dfd-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8\" (UID: \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"
Nov 03 09:02:17.298046 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.297980    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ecc8182-a9fd-471f-a00d-2195f2757dfd-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8\" (UID: \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"
Nov 03 09:02:17.298279 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.298056    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ecc8182-a9fd-471f-a00d-2195f2757dfd-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8\" (UID: \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"
Nov 03 09:02:17.298279 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.298110    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ecc8182-a9fd-471f-a00d-2195f2757dfd-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8\" (UID: \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"
Nov 03 09:02:17.298279 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.298163    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ecc8182-a9fd-471f-a00d-2195f2757dfd-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8\" (UID: \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"
Nov 03 09:02:17.298414 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.298393    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ecc8182-a9fd-471f-a00d-2195f2757dfd-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8\" (UID: \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"
Nov 03 09:02:17.298506 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.298488    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ecc8182-a9fd-471f-a00d-2195f2757dfd-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8\" (UID: \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"
Nov 03 09:02:17.298923 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.298898    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ecc8182-a9fd-471f-a00d-2195f2757dfd-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8\" (UID: \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"
Nov 03 09:02:17.300493 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.300475    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ecc8182-a9fd-471f-a00d-2195f2757dfd-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8\" (UID: \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"
Nov 03 09:02:17.300730 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.300709    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ecc8182-a9fd-471f-a00d-2195f2757dfd-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8\" (UID: \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"
Nov 03 09:02:17.356957 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.356899    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ecc8182-a9fd-471f-a00d-2195f2757dfd-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8\" (UID: \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"
Nov 03 09:02:17.357214 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.357190    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfj87\" (UniqueName: \"kubernetes.io/projected/4ecc8182-a9fd-471f-a00d-2195f2757dfd-kube-api-access-sfj87\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8\" (UID: \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"
Nov 03 09:02:17.445434 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.445388    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"
Nov 03 09:02:17.690300 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.690261    4473 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 09:02:17.690711 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.690320    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6" podUID="5f53106b-dc09-41eb-ab1a-16cd3fa8de90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 09:02:17.881538 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:02:17.881186    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ecc8182_a9fd_471f_a00d_2195f2757dfd.slice/crio-1243d863bc9b41ab680d877842478c208ec344a7df190ab7ab416a35b6ba9c91 WatchSource:0}: Error finding container 1243d863bc9b41ab680d877842478c208ec344a7df190ab7ab416a35b6ba9c91: Status 404 returned error can't find the container with id 1243d863bc9b41ab680d877842478c208ec344a7df190ab7ab416a35b6ba9c91
Nov 03 09:02:17.882311 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.882290    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"]
Nov 03 09:02:17.883998 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.883970    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:02:17.884081 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.884029    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:02:17.884081 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.884058    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:02:17.953030 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:17.952933    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8" event={"ID":"4ecc8182-a9fd-471f-a00d-2195f2757dfd","Type":"ContainerStarted","Data":"1243d863bc9b41ab680d877842478c208ec344a7df190ab7ab416a35b6ba9c91"}
Nov 03 09:02:18.691032 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:18.690995    4473 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 09:02:18.691457 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:18.691052    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6" podUID="5f53106b-dc09-41eb-ab1a-16cd3fa8de90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 09:02:18.974193 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:18.974059    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8" event={"ID":"4ecc8182-a9fd-471f-a00d-2195f2757dfd","Type":"ContainerStarted","Data":"661b822d3eac4602afaa31eb245b7404f4f100f635b76ac856c0a0a28a4d4933"}
Nov 03 09:02:19.156974 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:19.155441    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-140e33df-7dvl7"]
Nov 03 09:02:19.161702 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:19.161674    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-140e33df-7dvl7"
Nov 03 09:02:19.190149 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:19.190098    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8" podStartSLOduration=2.190080428 podStartE2EDuration="2.190080428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:02:19.15443497 +0000 UTC m=+1363.423498456" watchObservedRunningTime="2026-11-03 09:02:19.190080428 +0000 UTC m=+1363.459143862"
Nov 03 09:02:19.209554 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:19.209522    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-140e33df-7dvl7"]
Nov 03 09:02:19.250377 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:19.250270    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:02:19.250377 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:19.250326    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:02:19.250615 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:19.250397    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:02:19.250615 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:19.250421    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:02:19.257437 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:19.257403    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s5vj\" (UniqueName: \"kubernetes.io/projected/21322b5d-c21e-4ecf-8990-c964de95e28e-kube-api-access-2s5vj\") pod \"maas-api-revoke-sub-140e33df-7dvl7\" (UID: \"21322b5d-c21e-4ecf-8990-c964de95e28e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-140e33df-7dvl7"
Nov 03 09:02:19.257636 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:19.257547    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21322b5d-c21e-4ecf-8990-c964de95e28e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-140e33df-7dvl7\" (UID: \"21322b5d-c21e-4ecf-8990-c964de95e28e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-140e33df-7dvl7"
Nov 03 09:02:19.257691 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:19.257658    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21322b5d-c21e-4ecf-8990-c964de95e28e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-140e33df-7dvl7\" (UID: \"21322b5d-c21e-4ecf-8990-c964de95e28e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-140e33df-7dvl7"
Nov 03 09:02:19.359166 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:19.359124    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s5vj\" (UniqueName: \"kubernetes.io/projected/21322b5d-c21e-4ecf-8990-c964de95e28e-kube-api-access-2s5vj\") pod \"maas-api-revoke-sub-140e33df-7dvl7\" (UID: \"21322b5d-c21e-4ecf-8990-c964de95e28e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-140e33df-7dvl7"
Nov 03 09:02:19.359166 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:19.359174    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21322b5d-c21e-4ecf-8990-c964de95e28e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-140e33df-7dvl7\" (UID: \"21322b5d-c21e-4ecf-8990-c964de95e28e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-140e33df-7dvl7"
Nov 03 09:02:19.359436 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:19.359210    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21322b5d-c21e-4ecf-8990-c964de95e28e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-140e33df-7dvl7\" (UID: \"21322b5d-c21e-4ecf-8990-c964de95e28e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-140e33df-7dvl7"
Nov 03 09:02:19.359860 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:19.359836    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21322b5d-c21e-4ecf-8990-c964de95e28e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-140e33df-7dvl7\" (UID: \"21322b5d-c21e-4ecf-8990-c964de95e28e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-140e33df-7dvl7"
Nov 03 09:02:19.361787 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:19.361762    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21322b5d-c21e-4ecf-8990-c964de95e28e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-140e33df-7dvl7\" (UID: \"21322b5d-c21e-4ecf-8990-c964de95e28e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-140e33df-7dvl7"
Nov 03 09:02:19.429979 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:19.428598    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s5vj\" (UniqueName: \"kubernetes.io/projected/21322b5d-c21e-4ecf-8990-c964de95e28e-kube-api-access-2s5vj\") pod \"maas-api-revoke-sub-140e33df-7dvl7\" (UID: \"21322b5d-c21e-4ecf-8990-c964de95e28e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-140e33df-7dvl7"
Nov 03 09:02:19.447410 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:19.447378    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"
Nov 03 09:02:19.449190 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:19.449161    4473 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 09:02:19.449346 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:19.449213    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8" podUID="4ecc8182-a9fd-471f-a00d-2195f2757dfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 09:02:19.477445 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:19.477401    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-140e33df-7dvl7"
Nov 03 09:02:19.690279 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:19.690239    4473 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 09:02:19.690448 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:19.690310    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6" podUID="5f53106b-dc09-41eb-ab1a-16cd3fa8de90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 09:02:19.885566 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:19.885536    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-140e33df-7dvl7"]
Nov 03 09:02:19.886559 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:02:19.886529    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21322b5d_c21e_4ecf_8990_c964de95e28e.slice/crio-ccbcf57c86505a205098bbce6d43bec8ffb2f6e166556cff9859fa509b2a7758 WatchSource:0}: Error finding container ccbcf57c86505a205098bbce6d43bec8ffb2f6e166556cff9859fa509b2a7758: Status 404 returned error can't find the container with id ccbcf57c86505a205098bbce6d43bec8ffb2f6e166556cff9859fa509b2a7758
Nov 03 09:02:19.981533 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:19.981499    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-140e33df-7dvl7" event={"ID":"21322b5d-c21e-4ecf-8990-c964de95e28e","Type":"ContainerStarted","Data":"ccbcf57c86505a205098bbce6d43bec8ffb2f6e166556cff9859fa509b2a7758"}
Nov 03 09:02:20.446540 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:20.446351    4473 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 09:02:20.446540 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:20.446413    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8" podUID="4ecc8182-a9fd-471f-a00d-2195f2757dfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 09:02:20.705023 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:20.704918    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"
Nov 03 09:02:20.705252 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:20.705235    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"
Nov 03 09:02:20.706394 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:20.706374    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"
Nov 03 09:02:20.992849 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:20.992762    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-140e33df-7dvl7" event={"ID":"21322b5d-c21e-4ecf-8990-c964de95e28e","Type":"ContainerStarted","Data":"fb0440613a69cc29295046b4c80b90c0d0f7b01e4c938db843f1b1ad6c0547b1"}
Nov 03 09:02:21.109770 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:21.109722    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-140e33df-7dvl7" podStartSLOduration=2.109707334 podStartE2EDuration="2.109707334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:02:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:02:21.108810522 +0000 UTC m=+1365.377873977" watchObservedRunningTime="2026-11-03 09:02:21.109707334 +0000 UTC m=+1365.378770772"
Nov 03 09:02:21.448972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:21.448126    4473 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 09:02:21.448972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:21.448185    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8" podUID="4ecc8182-a9fd-471f-a00d-2195f2757dfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 09:02:22.075973 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:22.074048    4473 generic.go:356] "Generic (PLEG): container finished" podID="21322b5d-c21e-4ecf-8990-c964de95e28e" containerID="fb0440613a69cc29295046b4c80b90c0d0f7b01e4c938db843f1b1ad6c0547b1" exitCode=0
Nov 03 09:02:22.075973 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:22.075203    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-140e33df-7dvl7" event={"ID":"21322b5d-c21e-4ecf-8990-c964de95e28e","Type":"ContainerDied","Data":"fb0440613a69cc29295046b4c80b90c0d0f7b01e4c938db843f1b1ad6c0547b1"}
Nov 03 09:02:22.469972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:22.468336    4473 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 09:02:22.469972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:22.468400    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8" podUID="4ecc8182-a9fd-471f-a00d-2195f2757dfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 09:02:23.446758 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:23.446655    4473 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 09:02:23.447380 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:23.447332    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8" podUID="4ecc8182-a9fd-471f-a00d-2195f2757dfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 09:02:23.492143 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:23.492086    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6" podUID="25c366b3-b919-462b-b69a-82ce9dbd7436" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:02:23.720632 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:23.720231    4473 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn" podUID="05a8a676-3011-4597-9b71-b70c3387e2b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:02:23.938975 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:23.924347    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx" podUID="8e84a474-def6-41dc-8516-5e5ccf42ff46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:02:24.010196 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:24.004290    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd" podUID="cf4cf2fc-612a-4f5c-a596-543036dde7e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:02:24.010196 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:24.005109    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh" podUID="00bab831-c667-415d-b6b0-81fff4d57762" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:02:24.089580 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:24.088060    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn" podUID="035b0461-2dfa-42f4-8ea7-2d1911cddca9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:02:24.225037 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:24.198679    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn" podUID="05a8a676-3011-4597-9b71-b70c3387e2b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 03 09:02:24.225037 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:24.208594    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm" podUID="60b1d7ef-16c6-4b6a-973b-4a251593eac7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:02:24.468025 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:24.463113    4473 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 09:02:24.480154 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:24.480113    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8" podUID="4ecc8182-a9fd-471f-a00d-2195f2757dfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 09:02:24.505777 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:24.505739    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24cd268f-8tzfp"]
Nov 03 09:02:24.512303 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:24.512271    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-140e33df-7dvl7"
Nov 03 09:02:24.555981 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:24.525101    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21322b5d-c21e-4ecf-8990-c964de95e28e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21322b5d-c21e-4ecf-8990-c964de95e28e-maas-api-openshift-ca\") pod \"21322b5d-c21e-4ecf-8990-c964de95e28e\" (UID: \"21322b5d-c21e-4ecf-8990-c964de95e28e\") "
Nov 03 09:02:24.556483 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:24.556442    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21322b5d-c21e-4ecf-8990-c964de95e28e-kube-api-access-2s5vj\" (UniqueName: \"kubernetes.io/projected/21322b5d-c21e-4ecf-8990-c964de95e28e-kube-api-access-2s5vj\") pod \"21322b5d-c21e-4ecf-8990-c964de95e28e\" (UID: \"21322b5d-c21e-4ecf-8990-c964de95e28e\") "
Nov 03 09:02:24.556713 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:24.556683    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21322b5d-c21e-4ecf-8990-c964de95e28e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21322b5d-c21e-4ecf-8990-c964de95e28e-maas-api-xks-ca\") pod \"21322b5d-c21e-4ecf-8990-c964de95e28e\" (UID: \"21322b5d-c21e-4ecf-8990-c964de95e28e\") "
Nov 03 09:02:24.587984 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:24.587171    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21322b5d-c21e-4ecf-8990-c964de95e28e-maas-api-openshift-ca" pod "21322b5d-c21e-4ecf-8990-c964de95e28e" (UID: "21322b5d-c21e-4ecf-8990-c964de95e28e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:02:24.595005 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:24.594277    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24cd268f-8tzfp"]
Nov 03 09:02:24.595005 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:24.594410    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cd268f-8tzfp"
Nov 03 09:02:24.598881 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:24.598841    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21322b5d-c21e-4ecf-8990-c964de95e28e-kube-api-access-2s5vj" pod "21322b5d-c21e-4ecf-8990-c964de95e28e" (UID: "21322b5d-c21e-4ecf-8990-c964de95e28e"). InnerVolumeSpecName "kube-api-access-2s5vj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:02:24.599417 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:24.599168    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21322b5d-c21e-4ecf-8990-c964de95e28e-maas-api-xks-ca" pod "21322b5d-c21e-4ecf-8990-c964de95e28e" (UID: "21322b5d-c21e-4ecf-8990-c964de95e28e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:02:24.663519 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:24.663484    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq444\" (UniqueName: \"kubernetes.io/projected/b69a3f01-23ee-4433-85ad-c3e5300e58ff-kube-api-access-vq444\") pod \"maas-api-revoke-sub-24cd268f-8tzfp\" (UID: \"b69a3f01-23ee-4433-85ad-c3e5300e58ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cd268f-8tzfp"
Nov 03 09:02:24.663804 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:24.663786    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b69a3f01-23ee-4433-85ad-c3e5300e58ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24cd268f-8tzfp\" (UID: \"b69a3f01-23ee-4433-85ad-c3e5300e58ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cd268f-8tzfp"
Nov 03 09:02:24.664014 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:24.663998    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b69a3f01-23ee-4433-85ad-c3e5300e58ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24cd268f-8tzfp\" (UID: \"b69a3f01-23ee-4433-85ad-c3e5300e58ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cd268f-8tzfp"
Nov 03 09:02:24.664279 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:24.664261    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s5vj\" (UniqueName: \"kubernetes.io/projected/21322b5d-c21e-4ecf-8990-c964de95e28e-kube-api-access-2s5vj\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:02:24.664387 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:24.664373    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21322b5d-c21e-4ecf-8990-c964de95e28e-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:02:24.664485 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:24.664473    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21322b5d-c21e-4ecf-8990-c964de95e28e-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:02:24.765228 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:24.765169    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b69a3f01-23ee-4433-85ad-c3e5300e58ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24cd268f-8tzfp\" (UID: \"b69a3f01-23ee-4433-85ad-c3e5300e58ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cd268f-8tzfp"
Nov 03 09:02:24.765399 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:24.765308    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq444\" (UniqueName: \"kubernetes.io/projected/b69a3f01-23ee-4433-85ad-c3e5300e58ff-kube-api-access-vq444\") pod \"maas-api-revoke-sub-24cd268f-8tzfp\" (UID: \"b69a3f01-23ee-4433-85ad-c3e5300e58ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cd268f-8tzfp"
Nov 03 09:02:24.765399 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:24.765367    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b69a3f01-23ee-4433-85ad-c3e5300e58ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24cd268f-8tzfp\" (UID: \"b69a3f01-23ee-4433-85ad-c3e5300e58ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cd268f-8tzfp"
Nov 03 09:02:24.766518 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:24.766221    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b69a3f01-23ee-4433-85ad-c3e5300e58ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24cd268f-8tzfp\" (UID: \"b69a3f01-23ee-4433-85ad-c3e5300e58ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cd268f-8tzfp"
Nov 03 09:02:24.768076 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:24.768050    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b69a3f01-23ee-4433-85ad-c3e5300e58ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24cd268f-8tzfp\" (UID: \"b69a3f01-23ee-4433-85ad-c3e5300e58ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cd268f-8tzfp"
Nov 03 09:02:24.839799 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:24.839765    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq444\" (UniqueName: \"kubernetes.io/projected/b69a3f01-23ee-4433-85ad-c3e5300e58ff-kube-api-access-vq444\") pod \"maas-api-revoke-sub-24cd268f-8tzfp\" (UID: \"b69a3f01-23ee-4433-85ad-c3e5300e58ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cd268f-8tzfp"
Nov 03 09:02:24.992294 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:24.992220    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cd268f-8tzfp"
Nov 03 09:02:25.397047 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:02:25.397014    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb69a3f01_23ee_4433_85ad_c3e5300e58ff.slice/crio-41aed28ffee0e11290ae12e70b6d038a4355e484d2b26fc2ea2977e50a193caa WatchSource:0}: Error finding container 41aed28ffee0e11290ae12e70b6d038a4355e484d2b26fc2ea2977e50a193caa: Status 404 returned error can't find the container with id 41aed28ffee0e11290ae12e70b6d038a4355e484d2b26fc2ea2977e50a193caa
Nov 03 09:02:25.397733 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:25.397636    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24cd268f-8tzfp"]
Nov 03 09:02:25.446336 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:25.446309    4473 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 09:02:25.446469 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:25.446377    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8" podUID="4ecc8182-a9fd-471f-a00d-2195f2757dfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 09:02:25.476768 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:25.476741    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-140e33df-7dvl7"
Nov 03 09:02:25.476768 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:25.476747    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-140e33df-7dvl7" event={"ID":"21322b5d-c21e-4ecf-8990-c964de95e28e","Type":"ContainerDied","Data":"ccbcf57c86505a205098bbce6d43bec8ffb2f6e166556cff9859fa509b2a7758"}
Nov 03 09:02:25.477299 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:25.476799    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccbcf57c86505a205098bbce6d43bec8ffb2f6e166556cff9859fa509b2a7758"
Nov 03 09:02:25.478243 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:25.478217    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cd268f-8tzfp" event={"ID":"b69a3f01-23ee-4433-85ad-c3e5300e58ff","Type":"ContainerStarted","Data":"41aed28ffee0e11290ae12e70b6d038a4355e484d2b26fc2ea2977e50a193caa"}
Nov 03 09:02:26.449997 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:26.449973    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"
Nov 03 09:02:26.450453 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:26.450429    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"
Nov 03 09:02:26.451049 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:26.451027    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"
Nov 03 09:02:26.485682 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:26.485652    4473 generic.go:356] "Generic (PLEG): container finished" podID="b69a3f01-23ee-4433-85ad-c3e5300e58ff" containerID="7f5a3db50f449d4ccfa73ff2f291294ccd1eade95b2c61a634902c23b195dc13" exitCode=0
Nov 03 09:02:26.486079 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:26.485764    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cd268f-8tzfp" event={"ID":"b69a3f01-23ee-4433-85ad-c3e5300e58ff","Type":"ContainerDied","Data":"7f5a3db50f449d4ccfa73ff2f291294ccd1eade95b2c61a634902c23b195dc13"}
Nov 03 09:02:27.641529 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:27.641501    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cd268f-8tzfp"
Nov 03 09:02:27.698232 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:27.698199    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b69a3f01-23ee-4433-85ad-c3e5300e58ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b69a3f01-23ee-4433-85ad-c3e5300e58ff-maas-api-openshift-ca\") pod \"b69a3f01-23ee-4433-85ad-c3e5300e58ff\" (UID: \"b69a3f01-23ee-4433-85ad-c3e5300e58ff\") "
Nov 03 09:02:27.698402 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:27.698260    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b69a3f01-23ee-4433-85ad-c3e5300e58ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b69a3f01-23ee-4433-85ad-c3e5300e58ff-maas-api-xks-ca\") pod \"b69a3f01-23ee-4433-85ad-c3e5300e58ff\" (UID: \"b69a3f01-23ee-4433-85ad-c3e5300e58ff\") "
Nov 03 09:02:27.698402 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:27.698396    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b69a3f01-23ee-4433-85ad-c3e5300e58ff-kube-api-access-vq444\" (UniqueName: \"kubernetes.io/projected/b69a3f01-23ee-4433-85ad-c3e5300e58ff-kube-api-access-vq444\") pod \"b69a3f01-23ee-4433-85ad-c3e5300e58ff\" (UID: \"b69a3f01-23ee-4433-85ad-c3e5300e58ff\") "
Nov 03 09:02:27.698609 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:27.698547    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b69a3f01-23ee-4433-85ad-c3e5300e58ff-maas-api-openshift-ca" pod "b69a3f01-23ee-4433-85ad-c3e5300e58ff" (UID: "b69a3f01-23ee-4433-85ad-c3e5300e58ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:02:27.698752 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:27.698732    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b69a3f01-23ee-4433-85ad-c3e5300e58ff-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:02:27.700737 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:27.700713    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b69a3f01-23ee-4433-85ad-c3e5300e58ff-maas-api-xks-ca" pod "b69a3f01-23ee-4433-85ad-c3e5300e58ff" (UID: "b69a3f01-23ee-4433-85ad-c3e5300e58ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:02:27.700860 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:27.700795    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b69a3f01-23ee-4433-85ad-c3e5300e58ff-kube-api-access-vq444" pod "b69a3f01-23ee-4433-85ad-c3e5300e58ff" (UID: "b69a3f01-23ee-4433-85ad-c3e5300e58ff"). InnerVolumeSpecName "kube-api-access-vq444". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:02:27.800592 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:27.800512    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b69a3f01-23ee-4433-85ad-c3e5300e58ff-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:02:27.800592 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:27.800548    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq444\" (UniqueName: \"kubernetes.io/projected/b69a3f01-23ee-4433-85ad-c3e5300e58ff-kube-api-access-vq444\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:02:28.464804 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:28.464769    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5"]
Nov 03 09:02:28.465342 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:28.465327    4473 cpu_manager.go:414] "containerMap: removing container" podUID="21322b5d-c21e-4ecf-8990-c964de95e28e" containerName="revoke-keys"
Nov 03 09:02:28.465425 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:28.465344    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21322b5d-c21e-4ecf-8990-c964de95e28e" containerName="revoke-keys"
Nov 03 09:02:28.465425 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:28.465356    4473 cpu_manager.go:414] "containerMap: removing container" podUID="b69a3f01-23ee-4433-85ad-c3e5300e58ff" containerName="revoke-keys"
Nov 03 09:02:28.465425 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:28.465361    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b69a3f01-23ee-4433-85ad-c3e5300e58ff" containerName="revoke-keys"
Nov 03 09:02:28.465523 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:28.465441    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="21322b5d-c21e-4ecf-8990-c964de95e28e" containerName="revoke-keys"
Nov 03 09:02:28.465523 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:28.465449    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b69a3f01-23ee-4433-85ad-c3e5300e58ff" containerName="revoke-keys"
Nov 03 09:02:28.468777 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:28.468762    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5"
Nov 03 09:02:28.502124 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:28.502092    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cd268f-8tzfp"
Nov 03 09:02:28.502279 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:28.502124    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cd268f-8tzfp" event={"ID":"b69a3f01-23ee-4433-85ad-c3e5300e58ff","Type":"ContainerDied","Data":"41aed28ffee0e11290ae12e70b6d038a4355e484d2b26fc2ea2977e50a193caa"}
Nov 03 09:02:28.502279 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:28.502176    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41aed28ffee0e11290ae12e70b6d038a4355e484d2b26fc2ea2977e50a193caa"
Nov 03 09:02:28.509250 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:28.509216    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e42501e-d39d-4ae1-91c7-76c3c171538e-tmp-dir\") pod \"distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5\" (UID: \"2e42501e-d39d-4ae1-91c7-76c3c171538e\") " pod="e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5"
Nov 03 09:02:28.509396 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:28.509375    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e42501e-d39d-4ae1-91c7-76c3c171538e-home\") pod \"distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5\" (UID: \"2e42501e-d39d-4ae1-91c7-76c3c171538e\") " pod="e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5"
Nov 03 09:02:28.509438 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:28.509405    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e42501e-d39d-4ae1-91c7-76c3c171538e-tls-certs\") pod \"distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5\" (UID: \"2e42501e-d39d-4ae1-91c7-76c3c171538e\") " pod="e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5"
Nov 03 09:02:28.509438 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:28.509424    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw4mh\" (UniqueName: \"kubernetes.io/projected/2e42501e-d39d-4ae1-91c7-76c3c171538e-kube-api-access-zw4mh\") pod \"distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5\" (UID: \"2e42501e-d39d-4ae1-91c7-76c3c171538e\") " pod="e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5"
Nov 03 09:02:28.509519 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:28.509505    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e42501e-d39d-4ae1-91c7-76c3c171538e-model-cache\") pod \"distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5\" (UID: \"2e42501e-d39d-4ae1-91c7-76c3c171538e\") " pod="e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5"
Nov 03 09:02:28.509631 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:28.509619    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e42501e-d39d-4ae1-91c7-76c3c171538e-dshm\") pod \"distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5\" (UID: \"2e42501e-d39d-4ae1-91c7-76c3c171538e\") " pod="e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5"
Nov 03 09:02:28.527165 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:28.527136    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5"]
Nov 03 09:02:28.573372 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:28.573343    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-18f9da\"/\"distinct-w4-18f9da-kserve-self-signed-certs\""
Nov 03 09:02:28.633735 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:28.633694    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e42501e-d39d-4ae1-91c7-76c3c171538e-dshm\") pod \"distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5\" (UID: \"2e42501e-d39d-4ae1-91c7-76c3c171538e\") " pod="e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5"
Nov 03 09:02:28.633991 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:28.633968    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e42501e-d39d-4ae1-91c7-76c3c171538e-tmp-dir\") pod \"distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5\" (UID: \"2e42501e-d39d-4ae1-91c7-76c3c171538e\") " pod="e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5"
Nov 03 09:02:28.634227 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:28.634210    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e42501e-d39d-4ae1-91c7-76c3c171538e-home\") pod \"distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5\" (UID: \"2e42501e-d39d-4ae1-91c7-76c3c171538e\") " pod="e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5"
Nov 03 09:02:28.634337 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:28.634324    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e42501e-d39d-4ae1-91c7-76c3c171538e-tls-certs\") pod \"distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5\" (UID: \"2e42501e-d39d-4ae1-91c7-76c3c171538e\") " pod="e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5"
Nov 03 09:02:28.634471 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:28.634437    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw4mh\" (UniqueName: \"kubernetes.io/projected/2e42501e-d39d-4ae1-91c7-76c3c171538e-kube-api-access-zw4mh\") pod \"distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5\" (UID: \"2e42501e-d39d-4ae1-91c7-76c3c171538e\") " pod="e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5"
Nov 03 09:02:28.634597 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:28.634583    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e42501e-d39d-4ae1-91c7-76c3c171538e-model-cache\") pod \"distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5\" (UID: \"2e42501e-d39d-4ae1-91c7-76c3c171538e\") " pod="e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5"
Nov 03 09:02:28.643136 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:28.637380    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e42501e-d39d-4ae1-91c7-76c3c171538e-home\") pod \"distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5\" (UID: \"2e42501e-d39d-4ae1-91c7-76c3c171538e\") " pod="e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5"
Nov 03 09:02:28.643640 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:28.643308    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e42501e-d39d-4ae1-91c7-76c3c171538e-model-cache\") pod \"distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5\" (UID: \"2e42501e-d39d-4ae1-91c7-76c3c171538e\") " pod="e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5"
Nov 03 09:02:28.653978 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:28.644097    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e42501e-d39d-4ae1-91c7-76c3c171538e-tmp-dir\") pod \"distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5\" (UID: \"2e42501e-d39d-4ae1-91c7-76c3c171538e\") " pod="e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5"
Nov 03 09:02:28.654220 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:28.652880    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e42501e-d39d-4ae1-91c7-76c3c171538e-tls-certs\") pod \"distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5\" (UID: \"2e42501e-d39d-4ae1-91c7-76c3c171538e\") " pod="e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5"
Nov 03 09:02:28.668553 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:28.668515    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e42501e-d39d-4ae1-91c7-76c3c171538e-dshm\") pod \"distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5\" (UID: \"2e42501e-d39d-4ae1-91c7-76c3c171538e\") " pod="e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5"
Nov 03 09:02:28.704712 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:28.704667    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw4mh\" (UniqueName: \"kubernetes.io/projected/2e42501e-d39d-4ae1-91c7-76c3c171538e-kube-api-access-zw4mh\") pod \"distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5\" (UID: \"2e42501e-d39d-4ae1-91c7-76c3c171538e\") " pod="e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5"
Nov 03 09:02:28.780848 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:28.780759    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5"
Nov 03 09:02:29.274329 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:29.274308    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5"]
Nov 03 09:02:29.274751 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:02:29.274725    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e42501e_d39d_4ae1_91c7_76c3c171538e.slice/crio-66115fc418a0df9ccb64e9712d438e688b99d36801b9636615504d757f287565 WatchSource:0}: Error finding container 66115fc418a0df9ccb64e9712d438e688b99d36801b9636615504d757f287565: Status 404 returned error can't find the container with id 66115fc418a0df9ccb64e9712d438e688b99d36801b9636615504d757f287565
Nov 03 09:02:29.510796 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:29.510761    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5" event={"ID":"2e42501e-d39d-4ae1-91c7-76c3c171538e","Type":"ContainerStarted","Data":"804a116da52cdfcc4d7e8f31ff5aae521e8c0d7291a469edb1b72c1fa18a1a4c"}
Nov 03 09:02:29.510984 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:29.510807    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5" event={"ID":"2e42501e-d39d-4ae1-91c7-76c3c171538e","Type":"ContainerStarted","Data":"66115fc418a0df9ccb64e9712d438e688b99d36801b9636615504d757f287565"}
Nov 03 09:02:29.627586 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:29.627533    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5" podStartSLOduration=1.627518611 podStartE2EDuration="1.627518611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:02:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:02:29.626421587 +0000 UTC m=+1373.895485045" watchObservedRunningTime="2026-11-03 09:02:29.627518611 +0000 UTC m=+1373.896582049"
Nov 03 09:02:29.781090 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:29.781055    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5"
Nov 03 09:02:33.056938 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:33.056909    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34957a84-mpjdq"]
Nov 03 09:02:33.106825 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:33.106796    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34957a84-mpjdq"]
Nov 03 09:02:33.869315 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:33.869277    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9c34e5b-23a9-4978-998a-26125e56212e" path="/var/lib/kubelet/pods/b9c34e5b-23a9-4978-998a-26125e56212e/volumes"
Nov 03 09:02:36.214787 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:36.214752    4473 scope.go:122] "RemoveContainer" containerID="5e9ae64e0a5951d47c2ee6766f9b9e09868da6684a87dc34cffebd8f61ac8f0e"
Nov 03 09:02:36.224460 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:36.224440    4473 scope.go:122] "RemoveContainer" containerID="50174e5684c2820cd1a902c14fa5b978f853c0b5b715ab7be9e6b50512c7b7c2"
Nov 03 09:02:36.233140 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:36.233117    4473 scope.go:122] "RemoveContainer" containerID="47f8b4007463815a7d201a2a538af86097763f6672699381a8f8ed835143019e"
Nov 03 09:02:36.241558 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:36.241542    4473 scope.go:122] "RemoveContainer" containerID="9da526e550e9ec00029bc8df226fd49c32c5c5cd525249659a65c40b9035ad43"
Nov 03 09:02:36.282107 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:36.282081    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:02:38.785421 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:38.785378    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5"
Nov 03 09:02:39.426111 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:39.426075    4473 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 09:02:39.426449 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:39.426396    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9" podUID="bddb43dc-284b-475c-8f28-d8b320be94b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:02:39.536971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:39.536434    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5"
Nov 03 09:02:39.886083 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:39.886053    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5"
Nov 03 09:02:43.579139 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:43.579106    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d06c8459-2mgg9"]
Nov 03 09:02:43.584717 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:43.584691    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06c8459-2mgg9"
Nov 03 09:02:43.641896 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:43.641869    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d06c8459-2mgg9"]
Nov 03 09:02:43.684789 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:43.684758    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7cd80b7-fa65-416a-9221-c9f101e33757-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d06c8459-2mgg9\" (UID: \"a7cd80b7-fa65-416a-9221-c9f101e33757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06c8459-2mgg9"
Nov 03 09:02:43.684969 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:43.684838    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7cd80b7-fa65-416a-9221-c9f101e33757-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d06c8459-2mgg9\" (UID: \"a7cd80b7-fa65-416a-9221-c9f101e33757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06c8459-2mgg9"
Nov 03 09:02:43.684969 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:43.684928    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnw2x\" (UniqueName: \"kubernetes.io/projected/a7cd80b7-fa65-416a-9221-c9f101e33757-kube-api-access-tnw2x\") pod \"maas-api-revoke-sub-d06c8459-2mgg9\" (UID: \"a7cd80b7-fa65-416a-9221-c9f101e33757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06c8459-2mgg9"
Nov 03 09:02:43.687786 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:43.687769    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:02:43.687844 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:43.687792    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:02:43.687844 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:43.687819    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:02:43.688453 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:43.688437    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:02:43.785701 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:43.785675    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7cd80b7-fa65-416a-9221-c9f101e33757-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d06c8459-2mgg9\" (UID: \"a7cd80b7-fa65-416a-9221-c9f101e33757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06c8459-2mgg9"
Nov 03 09:02:43.785863 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:43.785759    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7cd80b7-fa65-416a-9221-c9f101e33757-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d06c8459-2mgg9\" (UID: \"a7cd80b7-fa65-416a-9221-c9f101e33757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06c8459-2mgg9"
Nov 03 09:02:43.785863 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:43.785783    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnw2x\" (UniqueName: \"kubernetes.io/projected/a7cd80b7-fa65-416a-9221-c9f101e33757-kube-api-access-tnw2x\") pod \"maas-api-revoke-sub-d06c8459-2mgg9\" (UID: \"a7cd80b7-fa65-416a-9221-c9f101e33757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06c8459-2mgg9"
Nov 03 09:02:43.786342 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:43.786322    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7cd80b7-fa65-416a-9221-c9f101e33757-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d06c8459-2mgg9\" (UID: \"a7cd80b7-fa65-416a-9221-c9f101e33757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06c8459-2mgg9"
Nov 03 09:02:43.788287 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:43.788261    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7cd80b7-fa65-416a-9221-c9f101e33757-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d06c8459-2mgg9\" (UID: \"a7cd80b7-fa65-416a-9221-c9f101e33757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06c8459-2mgg9"
Nov 03 09:02:43.845404 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:43.845330    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnw2x\" (UniqueName: \"kubernetes.io/projected/a7cd80b7-fa65-416a-9221-c9f101e33757-kube-api-access-tnw2x\") pod \"maas-api-revoke-sub-d06c8459-2mgg9\" (UID: \"a7cd80b7-fa65-416a-9221-c9f101e33757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06c8459-2mgg9"
Nov 03 09:02:43.900194 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:43.900160    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06c8459-2mgg9"
Nov 03 09:02:44.283095 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:44.283067    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d06c8459-2mgg9"]
Nov 03 09:02:44.283922 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:02:44.283894    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7cd80b7_fa65_416a_9221_c9f101e33757.slice/crio-5637ece0229522d3455c74a80e6ccb03adca95e06c3428054698ab38847dab67 WatchSource:0}: Error finding container 5637ece0229522d3455c74a80e6ccb03adca95e06c3428054698ab38847dab67: Status 404 returned error can't find the container with id 5637ece0229522d3455c74a80e6ccb03adca95e06c3428054698ab38847dab67
Nov 03 09:02:44.763628 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:44.763592    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06c8459-2mgg9" event={"ID":"a7cd80b7-fa65-416a-9221-c9f101e33757","Type":"ContainerStarted","Data":"1159eac4f000d90cf6344eb0e9e010a6d2fc33881fb72b850918e15e43030d92"}
Nov 03 09:02:44.764029 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:44.763642    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06c8459-2mgg9" event={"ID":"a7cd80b7-fa65-416a-9221-c9f101e33757","Type":"ContainerStarted","Data":"5637ece0229522d3455c74a80e6ccb03adca95e06c3428054698ab38847dab67"}
Nov 03 09:02:44.884565 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:44.884502    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06c8459-2mgg9" podStartSLOduration=1.884481149 podStartE2EDuration="1.884481149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:02:44.880387345 +0000 UTC m=+1389.149450799" watchObservedRunningTime="2026-11-03 09:02:44.884481149 +0000 UTC m=+1389.153544586"
Nov 03 09:02:45.771549 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:45.771514    4473 generic.go:356] "Generic (PLEG): container finished" podID="a7cd80b7-fa65-416a-9221-c9f101e33757" containerID="1159eac4f000d90cf6344eb0e9e010a6d2fc33881fb72b850918e15e43030d92" exitCode=0
Nov 03 09:02:45.771975 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:45.771599    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06c8459-2mgg9" event={"ID":"a7cd80b7-fa65-416a-9221-c9f101e33757","Type":"ContainerDied","Data":"1159eac4f000d90cf6344eb0e9e010a6d2fc33881fb72b850918e15e43030d92"}
Nov 03 09:02:46.925552 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:46.925529    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06c8459-2mgg9"
Nov 03 09:02:46.938686 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:02:46.938656    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:02:47.022224 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:47.022194    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7cd80b7-fa65-416a-9221-c9f101e33757-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7cd80b7-fa65-416a-9221-c9f101e33757-maas-api-xks-ca\") pod \"a7cd80b7-fa65-416a-9221-c9f101e33757\" (UID: \"a7cd80b7-fa65-416a-9221-c9f101e33757\") "
Nov 03 09:02:47.022398 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:47.022364    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7cd80b7-fa65-416a-9221-c9f101e33757-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7cd80b7-fa65-416a-9221-c9f101e33757-maas-api-openshift-ca\") pod \"a7cd80b7-fa65-416a-9221-c9f101e33757\" (UID: \"a7cd80b7-fa65-416a-9221-c9f101e33757\") "
Nov 03 09:02:47.022458 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:47.022445    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7cd80b7-fa65-416a-9221-c9f101e33757-kube-api-access-tnw2x\" (UniqueName: \"kubernetes.io/projected/a7cd80b7-fa65-416a-9221-c9f101e33757-kube-api-access-tnw2x\") pod \"a7cd80b7-fa65-416a-9221-c9f101e33757\" (UID: \"a7cd80b7-fa65-416a-9221-c9f101e33757\") "
Nov 03 09:02:47.022714 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:47.022689    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7cd80b7-fa65-416a-9221-c9f101e33757-maas-api-openshift-ca" pod "a7cd80b7-fa65-416a-9221-c9f101e33757" (UID: "a7cd80b7-fa65-416a-9221-c9f101e33757"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:02:47.022844 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:47.022828    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7cd80b7-fa65-416a-9221-c9f101e33757-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:02:47.024376 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:47.024344    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7cd80b7-fa65-416a-9221-c9f101e33757-maas-api-xks-ca" pod "a7cd80b7-fa65-416a-9221-c9f101e33757" (UID: "a7cd80b7-fa65-416a-9221-c9f101e33757"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:02:47.024465 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:47.024453    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7cd80b7-fa65-416a-9221-c9f101e33757-kube-api-access-tnw2x" pod "a7cd80b7-fa65-416a-9221-c9f101e33757" (UID: "a7cd80b7-fa65-416a-9221-c9f101e33757"). InnerVolumeSpecName "kube-api-access-tnw2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:02:47.123934 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:47.123901    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7cd80b7-fa65-416a-9221-c9f101e33757-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:02:47.123934 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:47.123929    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnw2x\" (UniqueName: \"kubernetes.io/projected/a7cd80b7-fa65-416a-9221-c9f101e33757-kube-api-access-tnw2x\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:02:47.795545 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:47.795515    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06c8459-2mgg9"
Nov 03 09:02:47.795728 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:47.795516    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06c8459-2mgg9" event={"ID":"a7cd80b7-fa65-416a-9221-c9f101e33757","Type":"ContainerDied","Data":"5637ece0229522d3455c74a80e6ccb03adca95e06c3428054698ab38847dab67"}
Nov 03 09:02:47.795728 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:47.795686    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5637ece0229522d3455c74a80e6ccb03adca95e06c3428054698ab38847dab67"
Nov 03 09:02:59.920874 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:59.918289    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn" podUID="035b0461-2dfa-42f4-8ea7-2d1911cddca9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:02:59.922300 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:59.921526    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx" podUID="8e84a474-def6-41dc-8516-5e5ccf42ff46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:02:59.950925 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:59.950845    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd" podUID="cf4cf2fc-612a-4f5c-a596-543036dde7e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:02:59.957404 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:02:59.956630    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh" podUID="00bab831-c667-415d-b6b0-81fff4d57762" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:03:00.155124 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:00.155078    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn" podUID="05a8a676-3011-4597-9b71-b70c3387e2b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:03:00.922081 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:00.922029    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx" podUID="8e84a474-def6-41dc-8516-5e5ccf42ff46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:03:00.922563 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:00.922435    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx"
Nov 03 09:03:01.417971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:01.412237    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6" podUID="25c366b3-b919-462b-b69a-82ce9dbd7436" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:03:01.488980 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:01.464018    4473 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh" podUID="00bab831-c667-415d-b6b0-81fff4d57762" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:03:01.716039 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:01.700143    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx"
Nov 03 09:03:01.794353 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:01.741130    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-whmmw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 09:03:01.794353 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:01.741190    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw" podUID="14dd6c95-a250-41c5-87c7-ae741f96d01a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:03:02.111682 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:02.111591    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj"]
Nov 03 09:03:02.139971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:02.132830    4473 cpu_manager.go:414] "containerMap: removing container" podUID="a7cd80b7-fa65-416a-9221-c9f101e33757" containerName="revoke-keys"
Nov 03 09:03:02.140636 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:02.140606    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7cd80b7-fa65-416a-9221-c9f101e33757" containerName="revoke-keys"
Nov 03 09:03:02.141315 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:02.141292    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7cd80b7-fa65-416a-9221-c9f101e33757" containerName="revoke-keys"
Nov 03 09:03:02.313164 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:02.313129    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj"]
Nov 03 09:03:02.352046 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:02.319110    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj"
Nov 03 09:03:02.458081 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:02.458046    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-910d50\"/\"distinct-2-w0-910d50-kserve-self-signed-certs\""
Nov 03 09:03:02.468458 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:02.464591    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6afd6431-8c5e-4a46-b505-270efaaacd1c-dshm\") pod \"distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj\" (UID: \"6afd6431-8c5e-4a46-b505-270efaaacd1c\") " pod="e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj"
Nov 03 09:03:02.468458 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:02.464650    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6afd6431-8c5e-4a46-b505-270efaaacd1c-tls-certs\") pod \"distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj\" (UID: \"6afd6431-8c5e-4a46-b505-270efaaacd1c\") " pod="e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj"
Nov 03 09:03:02.468458 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:02.464774    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6afd6431-8c5e-4a46-b505-270efaaacd1c-home\") pod \"distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj\" (UID: \"6afd6431-8c5e-4a46-b505-270efaaacd1c\") " pod="e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj"
Nov 03 09:03:02.468458 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:02.464885    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6afd6431-8c5e-4a46-b505-270efaaacd1c-tmp-dir\") pod \"distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj\" (UID: \"6afd6431-8c5e-4a46-b505-270efaaacd1c\") " pod="e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj"
Nov 03 09:03:02.468458 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:02.464936    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-544mx\" (UniqueName: \"kubernetes.io/projected/6afd6431-8c5e-4a46-b505-270efaaacd1c-kube-api-access-544mx\") pod \"distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj\" (UID: \"6afd6431-8c5e-4a46-b505-270efaaacd1c\") " pod="e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj"
Nov 03 09:03:02.468458 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:02.465011    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6afd6431-8c5e-4a46-b505-270efaaacd1c-model-cache\") pod \"distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj\" (UID: \"6afd6431-8c5e-4a46-b505-270efaaacd1c\") " pod="e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj"
Nov 03 09:03:02.578697 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:02.578655    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6afd6431-8c5e-4a46-b505-270efaaacd1c-home\") pod \"distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj\" (UID: \"6afd6431-8c5e-4a46-b505-270efaaacd1c\") " pod="e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj"
Nov 03 09:03:02.591247 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:02.591213    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6afd6431-8c5e-4a46-b505-270efaaacd1c-tmp-dir\") pod \"distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj\" (UID: \"6afd6431-8c5e-4a46-b505-270efaaacd1c\") " pod="e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj"
Nov 03 09:03:02.594092 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:02.594063    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-544mx\" (UniqueName: \"kubernetes.io/projected/6afd6431-8c5e-4a46-b505-270efaaacd1c-kube-api-access-544mx\") pod \"distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj\" (UID: \"6afd6431-8c5e-4a46-b505-270efaaacd1c\") " pod="e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj"
Nov 03 09:03:02.606123 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:02.606095    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6afd6431-8c5e-4a46-b505-270efaaacd1c-model-cache\") pod \"distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj\" (UID: \"6afd6431-8c5e-4a46-b505-270efaaacd1c\") " pod="e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj"
Nov 03 09:03:02.613979 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:02.606706    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6afd6431-8c5e-4a46-b505-270efaaacd1c-dshm\") pod \"distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj\" (UID: \"6afd6431-8c5e-4a46-b505-270efaaacd1c\") " pod="e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj"
Nov 03 09:03:02.614640 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:02.614616    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6afd6431-8c5e-4a46-b505-270efaaacd1c-tls-certs\") pod \"distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj\" (UID: \"6afd6431-8c5e-4a46-b505-270efaaacd1c\") " pod="e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj"
Nov 03 09:03:02.630972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:02.606610    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6afd6431-8c5e-4a46-b505-270efaaacd1c-model-cache\") pod \"distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj\" (UID: \"6afd6431-8c5e-4a46-b505-270efaaacd1c\") " pod="e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj"
Nov 03 09:03:02.631211 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:02.581202    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6afd6431-8c5e-4a46-b505-270efaaacd1c-home\") pod \"distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj\" (UID: \"6afd6431-8c5e-4a46-b505-270efaaacd1c\") " pod="e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj"
Nov 03 09:03:02.631326 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:02.591752    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6afd6431-8c5e-4a46-b505-270efaaacd1c-tmp-dir\") pod \"distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj\" (UID: \"6afd6431-8c5e-4a46-b505-270efaaacd1c\") " pod="e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj"
Nov 03 09:03:02.639798 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:02.639770    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6afd6431-8c5e-4a46-b505-270efaaacd1c-dshm\") pod \"distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj\" (UID: \"6afd6431-8c5e-4a46-b505-270efaaacd1c\") " pod="e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj"
Nov 03 09:03:02.683900 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:02.683869    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6afd6431-8c5e-4a46-b505-270efaaacd1c-tls-certs\") pod \"distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj\" (UID: \"6afd6431-8c5e-4a46-b505-270efaaacd1c\") " pod="e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj"
Nov 03 09:03:02.713268 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:02.712536    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-544mx\" (UniqueName: \"kubernetes.io/projected/6afd6431-8c5e-4a46-b505-270efaaacd1c-kube-api-access-544mx\") pod \"distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj\" (UID: \"6afd6431-8c5e-4a46-b505-270efaaacd1c\") " pod="e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj"
Nov 03 09:03:03.005984 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:03.005873    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj"
Nov 03 09:03:03.481701 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:03.481626    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj"]
Nov 03 09:03:03.777389 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:03.777296    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d7fd66-7cdbb5785c-zxv4z"]
Nov 03 09:03:03.784173 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:03.784141    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d7fd66-7cdbb5785c-zxv4z"
Nov 03 09:03:03.873132 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:03.873096    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68pws\" (UniqueName: \"kubernetes.io/projected/f80724ee-b824-4f48-a6af-4c9b8d4c163c-kube-api-access-68pws\") pod \"payload-processing-e2e-api-a-d7fd66-7cdbb5785c-zxv4z\" (UID: \"f80724ee-b824-4f48-a6af-4c9b8d4c163c\") " pod="openshift-ingress/payload-processing-e2e-api-a-d7fd66-7cdbb5785c-zxv4z"
Nov 03 09:03:03.873512 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:03.873491    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f80724ee-b824-4f48-a6af-4c9b8d4c163c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d7fd66-7cdbb5785c-zxv4z\" (UID: \"f80724ee-b824-4f48-a6af-4c9b8d4c163c\") " pod="openshift-ingress/payload-processing-e2e-api-a-d7fd66-7cdbb5785c-zxv4z"
Nov 03 09:03:03.893199 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:03.893148    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d7fd66\""
Nov 03 09:03:03.893199 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:03.893150    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d7fd66-dockercfg-4sdqv\""
Nov 03 09:03:03.948995 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:03.930754    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d7fd66-7cdbb5785c-zxv4z"]
Nov 03 09:03:03.974517 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:03.974486    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68pws\" (UniqueName: \"kubernetes.io/projected/f80724ee-b824-4f48-a6af-4c9b8d4c163c-kube-api-access-68pws\") pod \"payload-processing-e2e-api-a-d7fd66-7cdbb5785c-zxv4z\" (UID: \"f80724ee-b824-4f48-a6af-4c9b8d4c163c\") " pod="openshift-ingress/payload-processing-e2e-api-a-d7fd66-7cdbb5785c-zxv4z"
Nov 03 09:03:03.975086 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:03.975062    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f80724ee-b824-4f48-a6af-4c9b8d4c163c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d7fd66-7cdbb5785c-zxv4z\" (UID: \"f80724ee-b824-4f48-a6af-4c9b8d4c163c\") " pod="openshift-ingress/payload-processing-e2e-api-a-d7fd66-7cdbb5785c-zxv4z"
Nov 03 09:03:03.976046 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:03.976018    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f80724ee-b824-4f48-a6af-4c9b8d4c163c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d7fd66-7cdbb5785c-zxv4z\" (UID: \"f80724ee-b824-4f48-a6af-4c9b8d4c163c\") " pod="openshift-ingress/payload-processing-e2e-api-a-d7fd66-7cdbb5785c-zxv4z"
Nov 03 09:03:04.037431 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:04.037355    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68pws\" (UniqueName: \"kubernetes.io/projected/f80724ee-b824-4f48-a6af-4c9b8d4c163c-kube-api-access-68pws\") pod \"payload-processing-e2e-api-a-d7fd66-7cdbb5785c-zxv4z\" (UID: \"f80724ee-b824-4f48-a6af-4c9b8d4c163c\") " pod="openshift-ingress/payload-processing-e2e-api-a-d7fd66-7cdbb5785c-zxv4z"
Nov 03 09:03:04.168926 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:04.168897    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d7fd66-7cdbb5785c-zxv4z"
Nov 03 09:03:04.173629 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:04.173598    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj" event={"ID":"6afd6431-8c5e-4a46-b505-270efaaacd1c","Type":"ContainerStarted","Data":"a03b4f719ec45cdfa5aa271222818dff5f7438cbc4d891845f0cd8e7394b2c21"}
Nov 03 09:03:04.173731 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:04.173642    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj" event={"ID":"6afd6431-8c5e-4a46-b505-270efaaacd1c","Type":"ContainerStarted","Data":"825a4dba0e4a26e963fe5518bbac2d82628ed062b78136adb8c1cad324c0f5f8"}
Nov 03 09:03:04.294698 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:04.294604    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj" podStartSLOduration=3.294589265 podStartE2EDuration="3.294589265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:03:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:03:04.291203827 +0000 UTC m=+1408.560267262" watchObservedRunningTime="2026-11-03 09:03:04.294589265 +0000 UTC m=+1408.563652699"
Nov 03 09:03:04.579765 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:04.579733    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d7fd66-7cdbb5785c-zxv4z"]
Nov 03 09:03:04.580567 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:03:04.580541    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf80724ee_b824_4f48_a6af_4c9b8d4c163c.slice/crio-814edb53599792f484a6a2746900c4a4ee2124aaf2a2b71b64506292bfdae179 WatchSource:0}: Error finding container 814edb53599792f484a6a2746900c4a4ee2124aaf2a2b71b64506292bfdae179: Status 404 returned error can't find the container with id 814edb53599792f484a6a2746900c4a4ee2124aaf2a2b71b64506292bfdae179
Nov 03 09:03:05.010403 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:05.010374    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj"
Nov 03 09:03:05.181342 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:05.181309    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d7fd66-7cdbb5785c-zxv4z" event={"ID":"f80724ee-b824-4f48-a6af-4c9b8d4c163c","Type":"ContainerStarted","Data":"a17c73144985762de89de3488a53c25f100ae8a35370bfcefd4e506f307e921f"}
Nov 03 09:03:05.181547 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:05.181368    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d7fd66-7cdbb5785c-zxv4z" event={"ID":"f80724ee-b824-4f48-a6af-4c9b8d4c163c","Type":"ContainerStarted","Data":"814edb53599792f484a6a2746900c4a4ee2124aaf2a2b71b64506292bfdae179"}
Nov 03 09:03:05.181547 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:05.181511    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-d7fd66-7cdbb5785c-zxv4z"
Nov 03 09:03:05.310358 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:05.310255    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-d7fd66-7cdbb5785c-zxv4z" podStartSLOduration=2.310238311 podStartE2EDuration="2.310238311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:03:05.307827895 +0000 UTC m=+1409.576891352" watchObservedRunningTime="2026-11-03 09:03:05.310238311 +0000 UTC m=+1409.579301745"
Nov 03 09:03:06.248812 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:06.248776    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b3cbf19-6szrs"]
Nov 03 09:03:06.252613 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:06.252598    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b3cbf19-6szrs"
Nov 03 09:03:06.310703 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:06.310668    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b3cbf19-6szrs"]
Nov 03 09:03:06.326280 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:06.326243    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vms46\" (UniqueName: \"kubernetes.io/projected/26d61cea-a8f4-4856-85f7-f9461b2aaede-kube-api-access-vms46\") pod \"maas-api-revoke-sub-9b3cbf19-6szrs\" (UID: \"26d61cea-a8f4-4856-85f7-f9461b2aaede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b3cbf19-6szrs"
Nov 03 09:03:06.326463 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:06.326444    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26d61cea-a8f4-4856-85f7-f9461b2aaede-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b3cbf19-6szrs\" (UID: \"26d61cea-a8f4-4856-85f7-f9461b2aaede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b3cbf19-6szrs"
Nov 03 09:03:06.326536 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:06.326484    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26d61cea-a8f4-4856-85f7-f9461b2aaede-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b3cbf19-6szrs\" (UID: \"26d61cea-a8f4-4856-85f7-f9461b2aaede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b3cbf19-6szrs"
Nov 03 09:03:06.356679 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:06.356653    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:03:06.356679 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:06.356652    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:03:06.356977 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:06.356659    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:03:06.356977 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:06.356680    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:03:06.427374 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:06.427343    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26d61cea-a8f4-4856-85f7-f9461b2aaede-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b3cbf19-6szrs\" (UID: \"26d61cea-a8f4-4856-85f7-f9461b2aaede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b3cbf19-6szrs"
Nov 03 09:03:06.427586 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:06.427389    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26d61cea-a8f4-4856-85f7-f9461b2aaede-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b3cbf19-6szrs\" (UID: \"26d61cea-a8f4-4856-85f7-f9461b2aaede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b3cbf19-6szrs"
Nov 03 09:03:06.427586 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:06.427547    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vms46\" (UniqueName: \"kubernetes.io/projected/26d61cea-a8f4-4856-85f7-f9461b2aaede-kube-api-access-vms46\") pod \"maas-api-revoke-sub-9b3cbf19-6szrs\" (UID: \"26d61cea-a8f4-4856-85f7-f9461b2aaede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b3cbf19-6szrs"
Nov 03 09:03:06.428120 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:06.428098    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26d61cea-a8f4-4856-85f7-f9461b2aaede-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b3cbf19-6szrs\" (UID: \"26d61cea-a8f4-4856-85f7-f9461b2aaede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b3cbf19-6szrs"
Nov 03 09:03:06.429870 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:06.429846    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26d61cea-a8f4-4856-85f7-f9461b2aaede-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b3cbf19-6szrs\" (UID: \"26d61cea-a8f4-4856-85f7-f9461b2aaede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b3cbf19-6szrs"
Nov 03 09:03:06.487024 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:06.486992    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vms46\" (UniqueName: \"kubernetes.io/projected/26d61cea-a8f4-4856-85f7-f9461b2aaede-kube-api-access-vms46\") pod \"maas-api-revoke-sub-9b3cbf19-6szrs\" (UID: \"26d61cea-a8f4-4856-85f7-f9461b2aaede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b3cbf19-6szrs"
Nov 03 09:03:06.565678 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:06.565595    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b3cbf19-6szrs"
Nov 03 09:03:06.945327 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:06.945296    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b3cbf19-6szrs"]
Nov 03 09:03:06.945521 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:03:06.945371    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d61cea_a8f4_4856_85f7_f9461b2aaede.slice/crio-2e1a481b381a13d82fc55354cd2124a45fb154ba972cfab04dc6e8b077ab4035 WatchSource:0}: Error finding container 2e1a481b381a13d82fc55354cd2124a45fb154ba972cfab04dc6e8b077ab4035: Status 404 returned error can't find the container with id 2e1a481b381a13d82fc55354cd2124a45fb154ba972cfab04dc6e8b077ab4035
Nov 03 09:03:07.195301 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:07.195215    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b3cbf19-6szrs" event={"ID":"26d61cea-a8f4-4856-85f7-f9461b2aaede","Type":"ContainerStarted","Data":"2e1a481b381a13d82fc55354cd2124a45fb154ba972cfab04dc6e8b077ab4035"}
Nov 03 09:03:08.202398 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:08.202366    4473 generic.go:356] "Generic (PLEG): container finished" podID="26d61cea-a8f4-4856-85f7-f9461b2aaede" containerID="3eb952460cf10f107496d45b2f1c9511afb683d039d653f2dd23a489adf9c133" exitCode=0
Nov 03 09:03:08.202799 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:08.202459    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b3cbf19-6szrs" event={"ID":"26d61cea-a8f4-4856-85f7-f9461b2aaede","Type":"ContainerDied","Data":"3eb952460cf10f107496d45b2f1c9511afb683d039d653f2dd23a489adf9c133"}
Nov 03 09:03:09.356520 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:09.356496    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b3cbf19-6szrs"
Nov 03 09:03:09.462201 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:09.462111    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26d61cea-a8f4-4856-85f7-f9461b2aaede-kube-api-access-vms46\" (UniqueName: \"kubernetes.io/projected/26d61cea-a8f4-4856-85f7-f9461b2aaede-kube-api-access-vms46\") pod \"26d61cea-a8f4-4856-85f7-f9461b2aaede\" (UID: \"26d61cea-a8f4-4856-85f7-f9461b2aaede\") "
Nov 03 09:03:09.462376 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:09.462233    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26d61cea-a8f4-4856-85f7-f9461b2aaede-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26d61cea-a8f4-4856-85f7-f9461b2aaede-maas-api-xks-ca\") pod \"26d61cea-a8f4-4856-85f7-f9461b2aaede\" (UID: \"26d61cea-a8f4-4856-85f7-f9461b2aaede\") "
Nov 03 09:03:09.462376 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:09.462278    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26d61cea-a8f4-4856-85f7-f9461b2aaede-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26d61cea-a8f4-4856-85f7-f9461b2aaede-maas-api-openshift-ca\") pod \"26d61cea-a8f4-4856-85f7-f9461b2aaede\" (UID: \"26d61cea-a8f4-4856-85f7-f9461b2aaede\") "
Nov 03 09:03:09.462691 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:09.462670    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26d61cea-a8f4-4856-85f7-f9461b2aaede-maas-api-openshift-ca" pod "26d61cea-a8f4-4856-85f7-f9461b2aaede" (UID: "26d61cea-a8f4-4856-85f7-f9461b2aaede"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:03:09.464372 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:09.464343    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26d61cea-a8f4-4856-85f7-f9461b2aaede-kube-api-access-vms46" pod "26d61cea-a8f4-4856-85f7-f9461b2aaede" (UID: "26d61cea-a8f4-4856-85f7-f9461b2aaede"). InnerVolumeSpecName "kube-api-access-vms46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:03:09.464475 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:09.464420    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26d61cea-a8f4-4856-85f7-f9461b2aaede-maas-api-xks-ca" pod "26d61cea-a8f4-4856-85f7-f9461b2aaede" (UID: "26d61cea-a8f4-4856-85f7-f9461b2aaede"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:03:09.563964 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:09.563910    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vms46\" (UniqueName: \"kubernetes.io/projected/26d61cea-a8f4-4856-85f7-f9461b2aaede-kube-api-access-vms46\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:03:09.563964 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:09.563939    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26d61cea-a8f4-4856-85f7-f9461b2aaede-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:03:09.564173 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:09.563976    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26d61cea-a8f4-4856-85f7-f9461b2aaede-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:03:10.216716 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:10.216619    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b3cbf19-6szrs"
Nov 03 09:03:10.216716 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:10.216652    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b3cbf19-6szrs" event={"ID":"26d61cea-a8f4-4856-85f7-f9461b2aaede","Type":"ContainerDied","Data":"2e1a481b381a13d82fc55354cd2124a45fb154ba972cfab04dc6e8b077ab4035"}
Nov 03 09:03:10.216716 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:10.216702    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e1a481b381a13d82fc55354cd2124a45fb154ba972cfab04dc6e8b077ab4035"
Nov 03 09:03:12.227123 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:12.227094    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:03:13.011974 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:13.009565    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj"
Nov 03 09:03:13.167965 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:13.163366    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj"
Nov 03 09:03:13.262393 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:13.262304    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj"
Nov 03 09:03:16.200251 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:16.200001    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-d7fd66-7cdbb5785c-zxv4z"
Nov 03 09:03:16.937668 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:03:16.937633    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:03:36.280331 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:36.280302    4473 scope.go:122] "RemoveContainer" containerID="96caadeb61d84012f7504067985c1e8053513f31f7017ec06494501abfebe9e6"
Nov 03 09:03:46.989565 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:03:46.989532    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:03:47.649027 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:47.648994    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"]
Nov 03 09:03:47.649527 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:47.649513    4473 cpu_manager.go:414] "containerMap: removing container" podUID="26d61cea-a8f4-4856-85f7-f9461b2aaede" containerName="revoke-keys"
Nov 03 09:03:47.649570 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:47.649529    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26d61cea-a8f4-4856-85f7-f9461b2aaede" containerName="revoke-keys"
Nov 03 09:03:47.649631 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:47.649621    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="26d61cea-a8f4-4856-85f7-f9461b2aaede" containerName="revoke-keys"
Nov 03 09:03:47.654509 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:47.654494    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"
Nov 03 09:03:47.747799 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:47.747748    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"]
Nov 03 09:03:47.764145 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:47.764118    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03b06f04-3ec5-4d68-847c-1eb86a519e5f-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk\" (UID: \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"
Nov 03 09:03:47.764297 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:47.764159    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b06f04-3ec5-4d68-847c-1eb86a519e5f-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk\" (UID: \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"
Nov 03 09:03:47.764297 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:47.764185    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03b06f04-3ec5-4d68-847c-1eb86a519e5f-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk\" (UID: \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"
Nov 03 09:03:47.764297 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:47.764232    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03b06f04-3ec5-4d68-847c-1eb86a519e5f-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk\" (UID: \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"
Nov 03 09:03:47.764431 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:47.764320    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b06f04-3ec5-4d68-847c-1eb86a519e5f-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk\" (UID: \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"
Nov 03 09:03:47.764431 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:47.764351    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03b06f04-3ec5-4d68-847c-1eb86a519e5f-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk\" (UID: \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"
Nov 03 09:03:47.764431 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:47.764373    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7nqt\" (UniqueName: \"kubernetes.io/projected/03b06f04-3ec5-4d68-847c-1eb86a519e5f-kube-api-access-l7nqt\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk\" (UID: \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"
Nov 03 09:03:47.764521 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:47.764492    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03b06f04-3ec5-4d68-847c-1eb86a519e5f-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk\" (UID: \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"
Nov 03 09:03:47.764568 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:47.764537    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03b06f04-3ec5-4d68-847c-1eb86a519e5f-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk\" (UID: \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"
Nov 03 09:03:47.865246 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:47.865219    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7nqt\" (UniqueName: \"kubernetes.io/projected/03b06f04-3ec5-4d68-847c-1eb86a519e5f-kube-api-access-l7nqt\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk\" (UID: \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"
Nov 03 09:03:47.865424 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:47.865310    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03b06f04-3ec5-4d68-847c-1eb86a519e5f-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk\" (UID: \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"
Nov 03 09:03:47.865424 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:47.865364    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03b06f04-3ec5-4d68-847c-1eb86a519e5f-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk\" (UID: \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"
Nov 03 09:03:47.865424 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:47.865437    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03b06f04-3ec5-4d68-847c-1eb86a519e5f-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk\" (UID: \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"
Nov 03 09:03:47.865640 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:47.865481    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b06f04-3ec5-4d68-847c-1eb86a519e5f-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk\" (UID: \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"
Nov 03 09:03:47.865640 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:47.865523    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03b06f04-3ec5-4d68-847c-1eb86a519e5f-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk\" (UID: \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"
Nov 03 09:03:47.865640 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:47.865551    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03b06f04-3ec5-4d68-847c-1eb86a519e5f-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk\" (UID: \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"
Nov 03 09:03:47.865788 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:47.865635    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b06f04-3ec5-4d68-847c-1eb86a519e5f-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk\" (UID: \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"
Nov 03 09:03:47.865788 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:47.865671    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03b06f04-3ec5-4d68-847c-1eb86a519e5f-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk\" (UID: \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"
Nov 03 09:03:47.865908 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:47.865795    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03b06f04-3ec5-4d68-847c-1eb86a519e5f-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk\" (UID: \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"
Nov 03 09:03:47.865908 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:47.865878    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b06f04-3ec5-4d68-847c-1eb86a519e5f-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk\" (UID: \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"
Nov 03 09:03:47.866070 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:47.865980    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03b06f04-3ec5-4d68-847c-1eb86a519e5f-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk\" (UID: \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"
Nov 03 09:03:47.866070 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:47.865994    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b06f04-3ec5-4d68-847c-1eb86a519e5f-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk\" (UID: \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"
Nov 03 09:03:47.866452 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:47.866430    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03b06f04-3ec5-4d68-847c-1eb86a519e5f-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk\" (UID: \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"
Nov 03 09:03:47.867990 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:47.867963    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03b06f04-3ec5-4d68-847c-1eb86a519e5f-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk\" (UID: \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"
Nov 03 09:03:47.868085 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:47.868002    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03b06f04-3ec5-4d68-847c-1eb86a519e5f-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk\" (UID: \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"
Nov 03 09:03:47.926311 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:47.926242    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03b06f04-3ec5-4d68-847c-1eb86a519e5f-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk\" (UID: \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"
Nov 03 09:03:47.926311 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:47.926269    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7nqt\" (UniqueName: \"kubernetes.io/projected/03b06f04-3ec5-4d68-847c-1eb86a519e5f-kube-api-access-l7nqt\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk\" (UID: \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"
Nov 03 09:03:47.967334 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:47.967310    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"
Nov 03 09:03:48.406386 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:48.406348    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"]
Nov 03 09:03:48.406735 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:03:48.406483    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b06f04_3ec5_4d68_847c_1eb86a519e5f.slice/crio-5de532b1b5466065f5dc482c4bdc6e462a77ba557cf2f8a2e1c9ef1ba1b4295b WatchSource:0}: Error finding container 5de532b1b5466065f5dc482c4bdc6e462a77ba557cf2f8a2e1c9ef1ba1b4295b: Status 404 returned error can't find the container with id 5de532b1b5466065f5dc482c4bdc6e462a77ba557cf2f8a2e1c9ef1ba1b4295b
Nov 03 09:03:48.409005 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:48.408978    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:03:48.409079 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:48.409040    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:03:48.409079 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:48.409069    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:03:48.634971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:48.629663    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk" event={"ID":"03b06f04-3ec5-4d68-847c-1eb86a519e5f","Type":"ContainerStarted","Data":"be2af7270385a5614cb849717e3f56f2417f2ecea94a105abc5d5a0f704f84b0"}
Nov 03 09:03:48.634971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:48.629730    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk" event={"ID":"03b06f04-3ec5-4d68-847c-1eb86a519e5f","Type":"ContainerStarted","Data":"5de532b1b5466065f5dc482c4bdc6e462a77ba557cf2f8a2e1c9ef1ba1b4295b"}
Nov 03 09:03:48.755762 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:48.755669    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk" podStartSLOduration=1.755654765 podStartE2EDuration="1.755654765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:03:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:03:48.752644207 +0000 UTC m=+1453.021707665" watchObservedRunningTime="2026-11-03 09:03:48.755654765 +0000 UTC m=+1453.024718199"
Nov 03 09:03:48.964967 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:48.964921    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:03:48.968417 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:48.968395    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"
Nov 03 09:03:49.968995 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:49.968959    4473 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 09:03:49.969393 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:49.969022    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk" podUID="03b06f04-3ec5-4d68-847c-1eb86a519e5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 09:03:50.975628 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:50.975586    4473 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 09:03:50.980252 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:50.976162    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk" podUID="03b06f04-3ec5-4d68-847c-1eb86a519e5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 09:03:51.967863 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:51.967831    4473 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 09:03:51.968069 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:51.967892    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk" podUID="03b06f04-3ec5-4d68-847c-1eb86a519e5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 09:03:52.968576 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:52.968543    4473 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 09:03:52.969056 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:52.968594    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk" podUID="03b06f04-3ec5-4d68-847c-1eb86a519e5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 09:03:53.968401 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:53.968370    4473 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 09:03:53.968574 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:53.968426    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk" podUID="03b06f04-3ec5-4d68-847c-1eb86a519e5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 09:03:54.968361 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:54.968322    4473 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Nov 03 09:03:54.968839 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:54.968377    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk" podUID="03b06f04-3ec5-4d68-847c-1eb86a519e5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Nov 03 09:03:55.238554 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:55.238476    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc48c856-z8zwb"]
Nov 03 09:03:55.242425 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:55.242409    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc48c856-z8zwb"
Nov 03 09:03:55.303333 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:55.303303    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc48c856-z8zwb"]
Nov 03 09:03:55.326569 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:55.326536    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76a4c9d3-b074-41ee-a4f9-8ffab40c60e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc48c856-z8zwb\" (UID: \"76a4c9d3-b074-41ee-a4f9-8ffab40c60e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc48c856-z8zwb"
Nov 03 09:03:55.326730 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:55.326656    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqgm8\" (UniqueName: \"kubernetes.io/projected/76a4c9d3-b074-41ee-a4f9-8ffab40c60e9-kube-api-access-bqgm8\") pod \"maas-api-revoke-sub-bc48c856-z8zwb\" (UID: \"76a4c9d3-b074-41ee-a4f9-8ffab40c60e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc48c856-z8zwb"
Nov 03 09:03:55.326825 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:55.326811    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76a4c9d3-b074-41ee-a4f9-8ffab40c60e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc48c856-z8zwb\" (UID: \"76a4c9d3-b074-41ee-a4f9-8ffab40c60e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc48c856-z8zwb"
Nov 03 09:03:55.349027 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:55.348998    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:03:55.349027 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:55.348998    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:03:55.349216 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:55.348998    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:03:55.349216 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:55.349073    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:03:55.427774 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:55.427744    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76a4c9d3-b074-41ee-a4f9-8ffab40c60e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc48c856-z8zwb\" (UID: \"76a4c9d3-b074-41ee-a4f9-8ffab40c60e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc48c856-z8zwb"
Nov 03 09:03:55.427972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:55.427841    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqgm8\" (UniqueName: \"kubernetes.io/projected/76a4c9d3-b074-41ee-a4f9-8ffab40c60e9-kube-api-access-bqgm8\") pod \"maas-api-revoke-sub-bc48c856-z8zwb\" (UID: \"76a4c9d3-b074-41ee-a4f9-8ffab40c60e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc48c856-z8zwb"
Nov 03 09:03:55.428033 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:55.428017    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76a4c9d3-b074-41ee-a4f9-8ffab40c60e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc48c856-z8zwb\" (UID: \"76a4c9d3-b074-41ee-a4f9-8ffab40c60e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc48c856-z8zwb"
Nov 03 09:03:55.429029 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:55.429002    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76a4c9d3-b074-41ee-a4f9-8ffab40c60e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc48c856-z8zwb\" (UID: \"76a4c9d3-b074-41ee-a4f9-8ffab40c60e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc48c856-z8zwb"
Nov 03 09:03:55.430461 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:55.430438    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76a4c9d3-b074-41ee-a4f9-8ffab40c60e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc48c856-z8zwb\" (UID: \"76a4c9d3-b074-41ee-a4f9-8ffab40c60e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc48c856-z8zwb"
Nov 03 09:03:55.487647 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:55.487626    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqgm8\" (UniqueName: \"kubernetes.io/projected/76a4c9d3-b074-41ee-a4f9-8ffab40c60e9-kube-api-access-bqgm8\") pod \"maas-api-revoke-sub-bc48c856-z8zwb\" (UID: \"76a4c9d3-b074-41ee-a4f9-8ffab40c60e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc48c856-z8zwb"
Nov 03 09:03:55.557764 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:55.557704    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc48c856-z8zwb"
Nov 03 09:03:55.944408 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:03:55.944375    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76a4c9d3_b074_41ee_a4f9_8ffab40c60e9.slice/crio-1176784ef178d6d27b77d486969197ca750d53697d8f38ad07e951156993800b WatchSource:0}: Error finding container 1176784ef178d6d27b77d486969197ca750d53697d8f38ad07e951156993800b: Status 404 returned error can't find the container with id 1176784ef178d6d27b77d486969197ca750d53697d8f38ad07e951156993800b
Nov 03 09:03:55.946001 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:55.945970    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc48c856-z8zwb"]
Nov 03 09:03:56.730848 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:56.730814    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc48c856-z8zwb" event={"ID":"76a4c9d3-b074-41ee-a4f9-8ffab40c60e9","Type":"ContainerStarted","Data":"167c3b8bf7f67551ef77a0d519a8a47a9a7c09a2a554625aebd7f695fb0927e6"}
Nov 03 09:03:56.731391 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:56.730874    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc48c856-z8zwb" event={"ID":"76a4c9d3-b074-41ee-a4f9-8ffab40c60e9","Type":"ContainerStarted","Data":"1176784ef178d6d27b77d486969197ca750d53697d8f38ad07e951156993800b"}
Nov 03 09:03:56.848242 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:56.848179    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc48c856-z8zwb" podStartSLOduration=1.8481603309999999 podStartE2EDuration="1.848160331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:03:56.846614042 +0000 UTC m=+1461.115677528" watchObservedRunningTime="2026-11-03 09:03:56.848160331 +0000 UTC m=+1461.117223765"
Nov 03 09:03:57.009132 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:57.009051    4473 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 09:03:57.009295 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:57.009095    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk" podUID="03b06f04-3ec5-4d68-847c-1eb86a519e5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:03:57.048657 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:57.048606    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"
Nov 03 09:03:57.049065 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:57.049046    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"
Nov 03 09:03:57.049863 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:57.049841    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"
Nov 03 09:03:57.739012 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:57.738978    4473 generic.go:356] "Generic (PLEG): container finished" podID="76a4c9d3-b074-41ee-a4f9-8ffab40c60e9" containerID="167c3b8bf7f67551ef77a0d519a8a47a9a7c09a2a554625aebd7f695fb0927e6" exitCode=0
Nov 03 09:03:57.739423 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:57.739057    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc48c856-z8zwb" event={"ID":"76a4c9d3-b074-41ee-a4f9-8ffab40c60e9","Type":"ContainerDied","Data":"167c3b8bf7f67551ef77a0d519a8a47a9a7c09a2a554625aebd7f695fb0927e6"}
Nov 03 09:03:58.893035 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:58.893011    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc48c856-z8zwb"
Nov 03 09:03:58.974350 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:58.974322    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76a4c9d3-b074-41ee-a4f9-8ffab40c60e9-kube-api-access-bqgm8\" (UniqueName: \"kubernetes.io/projected/76a4c9d3-b074-41ee-a4f9-8ffab40c60e9-kube-api-access-bqgm8\") pod \"76a4c9d3-b074-41ee-a4f9-8ffab40c60e9\" (UID: \"76a4c9d3-b074-41ee-a4f9-8ffab40c60e9\") "
Nov 03 09:03:58.974521 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:58.974414    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76a4c9d3-b074-41ee-a4f9-8ffab40c60e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76a4c9d3-b074-41ee-a4f9-8ffab40c60e9-maas-api-openshift-ca\") pod \"76a4c9d3-b074-41ee-a4f9-8ffab40c60e9\" (UID: \"76a4c9d3-b074-41ee-a4f9-8ffab40c60e9\") "
Nov 03 09:03:58.974521 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:58.974472    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76a4c9d3-b074-41ee-a4f9-8ffab40c60e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76a4c9d3-b074-41ee-a4f9-8ffab40c60e9-maas-api-xks-ca\") pod \"76a4c9d3-b074-41ee-a4f9-8ffab40c60e9\" (UID: \"76a4c9d3-b074-41ee-a4f9-8ffab40c60e9\") "
Nov 03 09:03:58.974802 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:58.974773    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76a4c9d3-b074-41ee-a4f9-8ffab40c60e9-maas-api-openshift-ca" pod "76a4c9d3-b074-41ee-a4f9-8ffab40c60e9" (UID: "76a4c9d3-b074-41ee-a4f9-8ffab40c60e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:03:58.974962 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:58.974920    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76a4c9d3-b074-41ee-a4f9-8ffab40c60e9-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:03:58.976756 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:58.976735    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76a4c9d3-b074-41ee-a4f9-8ffab40c60e9-maas-api-xks-ca" pod "76a4c9d3-b074-41ee-a4f9-8ffab40c60e9" (UID: "76a4c9d3-b074-41ee-a4f9-8ffab40c60e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:03:58.976858 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:58.976782    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76a4c9d3-b074-41ee-a4f9-8ffab40c60e9-kube-api-access-bqgm8" pod "76a4c9d3-b074-41ee-a4f9-8ffab40c60e9" (UID: "76a4c9d3-b074-41ee-a4f9-8ffab40c60e9"). InnerVolumeSpecName "kube-api-access-bqgm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:03:59.075719 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:59.075639    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqgm8\" (UniqueName: \"kubernetes.io/projected/76a4c9d3-b074-41ee-a4f9-8ffab40c60e9-kube-api-access-bqgm8\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:03:59.075719 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:59.075670    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76a4c9d3-b074-41ee-a4f9-8ffab40c60e9-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:03:59.755535 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:59.755503    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc48c856-z8zwb"
Nov 03 09:03:59.755710 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:59.755558    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc48c856-z8zwb" event={"ID":"76a4c9d3-b074-41ee-a4f9-8ffab40c60e9","Type":"ContainerDied","Data":"1176784ef178d6d27b77d486969197ca750d53697d8f38ad07e951156993800b"}
Nov 03 09:03:59.755710 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:03:59.755649    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1176784ef178d6d27b77d486969197ca750d53697d8f38ad07e951156993800b"
Nov 03 09:04:00.864978 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:00.864922    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g"]
Nov 03 09:04:00.865444 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:00.865431    4473 cpu_manager.go:414] "containerMap: removing container" podUID="76a4c9d3-b074-41ee-a4f9-8ffab40c60e9" containerName="revoke-keys"
Nov 03 09:04:00.865488 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:00.865445    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76a4c9d3-b074-41ee-a4f9-8ffab40c60e9" containerName="revoke-keys"
Nov 03 09:04:00.865556 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:00.865547    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="76a4c9d3-b074-41ee-a4f9-8ffab40c60e9" containerName="revoke-keys"
Nov 03 09:04:00.868898 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:00.868883    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g"
Nov 03 09:04:00.972959 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:00.972919    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-18f9da\"/\"distinct-2-w4-18f9da-kserve-self-signed-certs\""
Nov 03 09:04:00.973793 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:00.973771    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g"]
Nov 03 09:04:01.007005 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:01.006973    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-tmp-dir\") pod \"distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g\" (UID: \"9e7f39b3-ce66-4b2d-abbc-8f078d527a2d\") " pod="e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g"
Nov 03 09:04:01.007356 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:01.007332    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-home\") pod \"distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g\" (UID: \"9e7f39b3-ce66-4b2d-abbc-8f078d527a2d\") " pod="e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g"
Nov 03 09:04:01.007506 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:01.007487    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlvth\" (UniqueName: \"kubernetes.io/projected/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-kube-api-access-rlvth\") pod \"distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g\" (UID: \"9e7f39b3-ce66-4b2d-abbc-8f078d527a2d\") " pod="e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g"
Nov 03 09:04:01.007729 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:01.007712    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-dshm\") pod \"distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g\" (UID: \"9e7f39b3-ce66-4b2d-abbc-8f078d527a2d\") " pod="e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g"
Nov 03 09:04:01.007909 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:01.007891    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-tls-certs\") pod \"distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g\" (UID: \"9e7f39b3-ce66-4b2d-abbc-8f078d527a2d\") " pod="e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g"
Nov 03 09:04:01.020541 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:01.019288    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-model-cache\") pod \"distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g\" (UID: \"9e7f39b3-ce66-4b2d-abbc-8f078d527a2d\") " pod="e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g"
Nov 03 09:04:01.122697 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:01.122670    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-tls-certs\") pod \"distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g\" (UID: \"9e7f39b3-ce66-4b2d-abbc-8f078d527a2d\") " pod="e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g"
Nov 03 09:04:01.123152 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:01.123125    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-model-cache\") pod \"distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g\" (UID: \"9e7f39b3-ce66-4b2d-abbc-8f078d527a2d\") " pod="e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g"
Nov 03 09:04:01.123288 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:01.123173    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-tmp-dir\") pod \"distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g\" (UID: \"9e7f39b3-ce66-4b2d-abbc-8f078d527a2d\") " pod="e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g"
Nov 03 09:04:01.123288 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:01.123278    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-home\") pod \"distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g\" (UID: \"9e7f39b3-ce66-4b2d-abbc-8f078d527a2d\") " pod="e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g"
Nov 03 09:04:01.123400 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:01.123307    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlvth\" (UniqueName: \"kubernetes.io/projected/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-kube-api-access-rlvth\") pod \"distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g\" (UID: \"9e7f39b3-ce66-4b2d-abbc-8f078d527a2d\") " pod="e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g"
Nov 03 09:04:01.123457 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:01.123441    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-dshm\") pod \"distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g\" (UID: \"9e7f39b3-ce66-4b2d-abbc-8f078d527a2d\") " pod="e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g"
Nov 03 09:04:01.124111 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:01.124087    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-tmp-dir\") pod \"distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g\" (UID: \"9e7f39b3-ce66-4b2d-abbc-8f078d527a2d\") " pod="e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g"
Nov 03 09:04:01.124352 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:01.124331    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-model-cache\") pod \"distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g\" (UID: \"9e7f39b3-ce66-4b2d-abbc-8f078d527a2d\") " pod="e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g"
Nov 03 09:04:01.124602 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:01.124583    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-home\") pod \"distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g\" (UID: \"9e7f39b3-ce66-4b2d-abbc-8f078d527a2d\") " pod="e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g"
Nov 03 09:04:01.129071 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:01.129045    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-tls-certs\") pod \"distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g\" (UID: \"9e7f39b3-ce66-4b2d-abbc-8f078d527a2d\") " pod="e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g"
Nov 03 09:04:01.130575 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:01.130549    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-dshm\") pod \"distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g\" (UID: \"9e7f39b3-ce66-4b2d-abbc-8f078d527a2d\") " pod="e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g"
Nov 03 09:04:01.183259 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:01.183227    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlvth\" (UniqueName: \"kubernetes.io/projected/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-kube-api-access-rlvth\") pod \"distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g\" (UID: \"9e7f39b3-ce66-4b2d-abbc-8f078d527a2d\") " pod="e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g"
Nov 03 09:04:01.481265 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:01.481187    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g"
Nov 03 09:04:01.863958 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:04:01.863913    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e7f39b3_ce66_4b2d_abbc_8f078d527a2d.slice/crio-e18c2d60643d0f54d6675596cc7507d7f178de0c3b1654df7247a88b430276f2 WatchSource:0}: Error finding container e18c2d60643d0f54d6675596cc7507d7f178de0c3b1654df7247a88b430276f2: Status 404 returned error can't find the container with id e18c2d60643d0f54d6675596cc7507d7f178de0c3b1654df7247a88b430276f2
Nov 03 09:04:01.870969 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:01.870931    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g"]
Nov 03 09:04:02.775681 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:02.775642    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g" event={"ID":"9e7f39b3-ce66-4b2d-abbc-8f078d527a2d","Type":"ContainerStarted","Data":"6cbfa3600b6ec31caa148e0864bef52b0805a33bb415ff10a049c4154e11b8f8"}
Nov 03 09:04:02.775875 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:02.775691    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g" event={"ID":"9e7f39b3-ce66-4b2d-abbc-8f078d527a2d","Type":"ContainerStarted","Data":"e18c2d60643d0f54d6675596cc7507d7f178de0c3b1654df7247a88b430276f2"}
Nov 03 09:04:02.893714 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:02.893660    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g" podStartSLOduration=2.893647629 podStartE2EDuration="2.893647629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:04:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:04:02.892598885 +0000 UTC m=+1467.161662344" watchObservedRunningTime="2026-11-03 09:04:02.893647629 +0000 UTC m=+1467.162711062"
Nov 03 09:04:03.481918 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:03.481887    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g"
Nov 03 09:04:07.910476 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:07.910442    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"]
Nov 03 09:04:07.914663 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:07.914641    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:04:07.996170 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:07.996121    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-204dfc-openshift-default-dockercfg-rjz9h\""
Nov 03 09:04:08.090743 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:08.090656    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"]
Nov 03 09:04:08.092241 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:08.091499    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/625c61e9-f360-4f60-9b21-885c4696aba3-workload-socket\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68\" (UID: \"625c61e9-f360-4f60-9b21-885c4696aba3\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:04:08.092484 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:08.092465    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/625c61e9-f360-4f60-9b21-885c4696aba3-workload-certs\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68\" (UID: \"625c61e9-f360-4f60-9b21-885c4696aba3\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:04:08.093062 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:08.093040    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/625c61e9-f360-4f60-9b21-885c4696aba3-istio-podinfo\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68\" (UID: \"625c61e9-f360-4f60-9b21-885c4696aba3\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:04:08.093292 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:08.093275    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/625c61e9-f360-4f60-9b21-885c4696aba3-istio-envoy\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68\" (UID: \"625c61e9-f360-4f60-9b21-885c4696aba3\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:04:08.093442 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:08.093426    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/625c61e9-f360-4f60-9b21-885c4696aba3-istio-token\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68\" (UID: \"625c61e9-f360-4f60-9b21-885c4696aba3\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:04:08.093617 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:08.093600    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/625c61e9-f360-4f60-9b21-885c4696aba3-istio-data\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68\" (UID: \"625c61e9-f360-4f60-9b21-885c4696aba3\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:04:08.103326 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:08.103301    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/625c61e9-f360-4f60-9b21-885c4696aba3-credential-socket\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68\" (UID: \"625c61e9-f360-4f60-9b21-885c4696aba3\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:04:08.103665 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:08.103644    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4rpk\" (UniqueName: \"kubernetes.io/projected/625c61e9-f360-4f60-9b21-885c4696aba3-kube-api-access-t4rpk\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68\" (UID: \"625c61e9-f360-4f60-9b21-885c4696aba3\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:04:08.105701 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:08.105679    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/625c61e9-f360-4f60-9b21-885c4696aba3-istiod-ca-cert\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68\" (UID: \"625c61e9-f360-4f60-9b21-885c4696aba3\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:04:08.216601 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:08.216522    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/625c61e9-f360-4f60-9b21-885c4696aba3-credential-socket\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68\" (UID: \"625c61e9-f360-4f60-9b21-885c4696aba3\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:04:08.217350 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:08.217329    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4rpk\" (UniqueName: \"kubernetes.io/projected/625c61e9-f360-4f60-9b21-885c4696aba3-kube-api-access-t4rpk\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68\" (UID: \"625c61e9-f360-4f60-9b21-885c4696aba3\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:04:08.217857 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:08.217240    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/625c61e9-f360-4f60-9b21-885c4696aba3-credential-socket\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68\" (UID: \"625c61e9-f360-4f60-9b21-885c4696aba3\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:04:08.218048 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:08.218032    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/625c61e9-f360-4f60-9b21-885c4696aba3-istiod-ca-cert\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68\" (UID: \"625c61e9-f360-4f60-9b21-885c4696aba3\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:04:08.218845 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:08.218716    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/625c61e9-f360-4f60-9b21-885c4696aba3-istiod-ca-cert\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68\" (UID: \"625c61e9-f360-4f60-9b21-885c4696aba3\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:04:08.219150 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:08.219131    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/625c61e9-f360-4f60-9b21-885c4696aba3-workload-socket\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68\" (UID: \"625c61e9-f360-4f60-9b21-885c4696aba3\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:04:08.219637 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:08.219535    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/625c61e9-f360-4f60-9b21-885c4696aba3-workload-socket\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68\" (UID: \"625c61e9-f360-4f60-9b21-885c4696aba3\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:04:08.219810 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:08.219792    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/625c61e9-f360-4f60-9b21-885c4696aba3-workload-certs\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68\" (UID: \"625c61e9-f360-4f60-9b21-885c4696aba3\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:04:08.220242 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:08.220142    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/625c61e9-f360-4f60-9b21-885c4696aba3-workload-certs\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68\" (UID: \"625c61e9-f360-4f60-9b21-885c4696aba3\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:04:08.220431 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:08.220411    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/625c61e9-f360-4f60-9b21-885c4696aba3-istio-podinfo\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68\" (UID: \"625c61e9-f360-4f60-9b21-885c4696aba3\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:04:08.221227 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:08.221210    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/625c61e9-f360-4f60-9b21-885c4696aba3-istio-envoy\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68\" (UID: \"625c61e9-f360-4f60-9b21-885c4696aba3\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:04:08.221714 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:08.221697    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/625c61e9-f360-4f60-9b21-885c4696aba3-istio-token\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68\" (UID: \"625c61e9-f360-4f60-9b21-885c4696aba3\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:04:08.222227 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:08.222204    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/625c61e9-f360-4f60-9b21-885c4696aba3-istio-data\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68\" (UID: \"625c61e9-f360-4f60-9b21-885c4696aba3\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:04:08.222709 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:08.222603    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/625c61e9-f360-4f60-9b21-885c4696aba3-istio-data\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68\" (UID: \"625c61e9-f360-4f60-9b21-885c4696aba3\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:04:08.288572 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:08.287126    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/625c61e9-f360-4f60-9b21-885c4696aba3-istio-envoy\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68\" (UID: \"625c61e9-f360-4f60-9b21-885c4696aba3\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:04:08.289756 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:08.289731    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/625c61e9-f360-4f60-9b21-885c4696aba3-istio-podinfo\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68\" (UID: \"625c61e9-f360-4f60-9b21-885c4696aba3\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:04:08.310904 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:08.310831    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4rpk\" (UniqueName: \"kubernetes.io/projected/625c61e9-f360-4f60-9b21-885c4696aba3-kube-api-access-t4rpk\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68\" (UID: \"625c61e9-f360-4f60-9b21-885c4696aba3\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:04:08.315812 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:08.315744    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/625c61e9-f360-4f60-9b21-885c4696aba3-istio-token\") pod \"e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68\" (UID: \"625c61e9-f360-4f60-9b21-885c4696aba3\") " pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:04:08.530714 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:08.530629    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:04:08.967896 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:08.967868    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"]
Nov 03 09:04:08.968318 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:04:08.968159    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod625c61e9_f360_4f60_9b21_885c4696aba3.slice/crio-1837a60ac251b2df674fbf46ee7c2a579ea3c0f74b80429f5ca258891f1d0d57 WatchSource:0}: Error finding container 1837a60ac251b2df674fbf46ee7c2a579ea3c0f74b80429f5ca258891f1d0d57: Status 404 returned error can't find the container with id 1837a60ac251b2df674fbf46ee7c2a579ea3c0f74b80429f5ca258891f1d0d57
Nov 03 09:04:08.971374 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:08.971336    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:04:08.971471 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:08.971422    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:04:08.971525 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:08.971467    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:04:09.831184 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:09.831141    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68" event={"ID":"625c61e9-f360-4f60-9b21-885c4696aba3","Type":"ContainerStarted","Data":"8e2707fcf56521b5ddf62f7183fd97098e64a10cd344cb52fecbaf9ac901a00e"}
Nov 03 09:04:09.831359 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:09.831197    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68" event={"ID":"625c61e9-f360-4f60-9b21-885c4696aba3","Type":"ContainerStarted","Data":"1837a60ac251b2df674fbf46ee7c2a579ea3c0f74b80429f5ca258891f1d0d57"}
Nov 03 09:04:10.530993 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:10.530960    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:04:10.537694 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:10.537665    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:04:10.708153 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:10.708094    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68" podStartSLOduration=3.708077117 podStartE2EDuration="3.708077117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:04:10.050118323 +0000 UTC m=+1474.319181756" watchObservedRunningTime="2026-11-03 09:04:10.708077117 +0000 UTC m=+1474.977140554"
Nov 03 09:04:10.838313 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:10.838234    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:04:10.839277 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:10.839258    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:04:11.482206 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:11.482181    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g"
Nov 03 09:04:11.494366 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:11.494336    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g"
Nov 03 09:04:11.855530 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:11.855451    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g"
Nov 03 09:04:16.937705 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:04:16.937613    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:04:17.941620 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:17.941590    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:04:26.651181 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:26.651140    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"]
Nov 03 09:04:26.655172 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:26.655154    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"
Nov 03 09:04:26.745445 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:26.745379    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"]
Nov 03 09:04:26.777973 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:26.764728    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e740caf-68ae-4528-8965-a1085396deca-credential-socket\") pod \"e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr\" (UID: \"6e740caf-68ae-4528-8965-a1085396deca\") " pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"
Nov 03 09:04:26.809217 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:26.809176    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e740caf-68ae-4528-8965-a1085396deca-istiod-ca-cert\") pod \"e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr\" (UID: \"6e740caf-68ae-4528-8965-a1085396deca\") " pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"
Nov 03 09:04:26.810144 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:26.810110    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e740caf-68ae-4528-8965-a1085396deca-istio-data\") pod \"e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr\" (UID: \"6e740caf-68ae-4528-8965-a1085396deca\") " pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"
Nov 03 09:04:26.810279 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:26.810188    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e740caf-68ae-4528-8965-a1085396deca-istio-token\") pod \"e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr\" (UID: \"6e740caf-68ae-4528-8965-a1085396deca\") " pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"
Nov 03 09:04:26.810352 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:26.810341    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e740caf-68ae-4528-8965-a1085396deca-istio-podinfo\") pod \"e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr\" (UID: \"6e740caf-68ae-4528-8965-a1085396deca\") " pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"
Nov 03 09:04:26.810407 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:26.810380    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8n42\" (UniqueName: \"kubernetes.io/projected/6e740caf-68ae-4528-8965-a1085396deca-kube-api-access-b8n42\") pod \"e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr\" (UID: \"6e740caf-68ae-4528-8965-a1085396deca\") " pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"
Nov 03 09:04:26.810485 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:26.810468    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e740caf-68ae-4528-8965-a1085396deca-istio-envoy\") pod \"e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr\" (UID: \"6e740caf-68ae-4528-8965-a1085396deca\") " pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"
Nov 03 09:04:26.810538 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:26.810504    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e740caf-68ae-4528-8965-a1085396deca-workload-certs\") pod \"e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr\" (UID: \"6e740caf-68ae-4528-8965-a1085396deca\") " pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"
Nov 03 09:04:26.810591 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:26.810559    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e740caf-68ae-4528-8965-a1085396deca-workload-socket\") pod \"e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr\" (UID: \"6e740caf-68ae-4528-8965-a1085396deca\") " pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"
Nov 03 09:04:26.810675 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:26.778389    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-b8fd34-openshift-default-dockercfg-s4v54\""
Nov 03 09:04:26.941392 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:26.930067    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e740caf-68ae-4528-8965-a1085396deca-workload-socket\") pod \"e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr\" (UID: \"6e740caf-68ae-4528-8965-a1085396deca\") " pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"
Nov 03 09:04:26.941392 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:26.930145    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e740caf-68ae-4528-8965-a1085396deca-credential-socket\") pod \"e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr\" (UID: \"6e740caf-68ae-4528-8965-a1085396deca\") " pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"
Nov 03 09:04:26.941392 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:26.930187    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e740caf-68ae-4528-8965-a1085396deca-istiod-ca-cert\") pod \"e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr\" (UID: \"6e740caf-68ae-4528-8965-a1085396deca\") " pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"
Nov 03 09:04:26.941392 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:26.930248    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e740caf-68ae-4528-8965-a1085396deca-istio-data\") pod \"e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr\" (UID: \"6e740caf-68ae-4528-8965-a1085396deca\") " pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"
Nov 03 09:04:26.941392 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:26.930301    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e740caf-68ae-4528-8965-a1085396deca-istio-token\") pod \"e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr\" (UID: \"6e740caf-68ae-4528-8965-a1085396deca\") " pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"
Nov 03 09:04:26.941392 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:26.930434    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e740caf-68ae-4528-8965-a1085396deca-istio-podinfo\") pod \"e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr\" (UID: \"6e740caf-68ae-4528-8965-a1085396deca\") " pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"
Nov 03 09:04:26.941392 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:26.930467    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8n42\" (UniqueName: \"kubernetes.io/projected/6e740caf-68ae-4528-8965-a1085396deca-kube-api-access-b8n42\") pod \"e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr\" (UID: \"6e740caf-68ae-4528-8965-a1085396deca\") " pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"
Nov 03 09:04:26.941392 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:26.930540    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e740caf-68ae-4528-8965-a1085396deca-istio-envoy\") pod \"e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr\" (UID: \"6e740caf-68ae-4528-8965-a1085396deca\") " pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"
Nov 03 09:04:26.941392 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:26.930567    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e740caf-68ae-4528-8965-a1085396deca-workload-certs\") pod \"e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr\" (UID: \"6e740caf-68ae-4528-8965-a1085396deca\") " pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"
Nov 03 09:04:26.941392 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:26.940337    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e740caf-68ae-4528-8965-a1085396deca-workload-socket\") pod \"e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr\" (UID: \"6e740caf-68ae-4528-8965-a1085396deca\") " pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"
Nov 03 09:04:26.941392 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:26.940591    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e740caf-68ae-4528-8965-a1085396deca-credential-socket\") pod \"e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr\" (UID: \"6e740caf-68ae-4528-8965-a1085396deca\") " pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"
Nov 03 09:04:26.941392 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:26.941231    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e740caf-68ae-4528-8965-a1085396deca-istiod-ca-cert\") pod \"e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr\" (UID: \"6e740caf-68ae-4528-8965-a1085396deca\") " pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"
Nov 03 09:04:26.942154 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:26.941457    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e740caf-68ae-4528-8965-a1085396deca-istio-data\") pod \"e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr\" (UID: \"6e740caf-68ae-4528-8965-a1085396deca\") " pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"
Nov 03 09:04:26.942926 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:26.942321    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e740caf-68ae-4528-8965-a1085396deca-workload-certs\") pod \"e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr\" (UID: \"6e740caf-68ae-4528-8965-a1085396deca\") " pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"
Nov 03 09:04:27.027618 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:27.027552    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e740caf-68ae-4528-8965-a1085396deca-istio-token\") pod \"e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr\" (UID: \"6e740caf-68ae-4528-8965-a1085396deca\") " pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"
Nov 03 09:04:27.030568 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:27.030526    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8n42\" (UniqueName: \"kubernetes.io/projected/6e740caf-68ae-4528-8965-a1085396deca-kube-api-access-b8n42\") pod \"e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr\" (UID: \"6e740caf-68ae-4528-8965-a1085396deca\") " pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"
Nov 03 09:04:27.048038 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:27.047974    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e740caf-68ae-4528-8965-a1085396deca-istio-podinfo\") pod \"e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr\" (UID: \"6e740caf-68ae-4528-8965-a1085396deca\") " pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"
Nov 03 09:04:27.051978 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:27.048819    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e740caf-68ae-4528-8965-a1085396deca-istio-envoy\") pod \"e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr\" (UID: \"6e740caf-68ae-4528-8965-a1085396deca\") " pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"
Nov 03 09:04:27.269591 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:27.269518    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"
Nov 03 09:04:27.719068 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:27.719038    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"]
Nov 03 09:04:27.719488 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:04:27.719303    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e740caf_68ae_4528_8965_a1085396deca.slice/crio-a390bbaf87b6025b59076870f9c78ace9e4243cee12d842bc5241f1ed12fbf7e WatchSource:0}: Error finding container a390bbaf87b6025b59076870f9c78ace9e4243cee12d842bc5241f1ed12fbf7e: Status 404 returned error can't find the container with id a390bbaf87b6025b59076870f9c78ace9e4243cee12d842bc5241f1ed12fbf7e
Nov 03 09:04:27.722112 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:27.722075    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:04:27.722200 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:27.722161    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:04:27.722242 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:27.722228    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:04:27.985648 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:27.985563    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr" event={"ID":"6e740caf-68ae-4528-8965-a1085396deca","Type":"ContainerStarted","Data":"88b7efd12000a2b1277ba153152f5a01be2ab91999f6116d182a19e1545b52c2"}
Nov 03 09:04:27.985648 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:27.985627    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr" event={"ID":"6e740caf-68ae-4528-8965-a1085396deca","Type":"ContainerStarted","Data":"a390bbaf87b6025b59076870f9c78ace9e4243cee12d842bc5241f1ed12fbf7e"}
Nov 03 09:04:28.104764 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:28.104701    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr" podStartSLOduration=2.104687761 podStartE2EDuration="2.104687761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:04:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:04:28.103786995 +0000 UTC m=+1492.372850456" watchObservedRunningTime="2026-11-03 09:04:28.104687761 +0000 UTC m=+1492.373751195"
Nov 03 09:04:28.272410 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:28.272324    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"
Nov 03 09:04:28.291000 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:28.290936    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"
Nov 03 09:04:28.991925 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:28.991895    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"
Nov 03 09:04:28.992745 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:28.992717    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"
Nov 03 09:04:34.686969 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:34.677648    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b8fd34-6d55957755-c955l"]
Nov 03 09:04:34.736226 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:34.728275    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b8fd34-6d55957755-c955l"
Nov 03 09:04:34.739699 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:34.739664    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b8fd34-6d55957755-c955l"]
Nov 03 09:04:34.785795 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:34.763768    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83047914-9b8f-4ea3-9b3e-caea9f8dcaa1-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b8fd34-6d55957755-c955l\" (UID: \"83047914-9b8f-4ea3-9b3e-caea9f8dcaa1\") " pod="openshift-ingress/payload-processing-e2e-api-b-b8fd34-6d55957755-c955l"
Nov 03 09:04:34.823268 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:34.815349    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8xn7\" (UniqueName: \"kubernetes.io/projected/83047914-9b8f-4ea3-9b3e-caea9f8dcaa1-kube-api-access-p8xn7\") pod \"payload-processing-e2e-api-b-b8fd34-6d55957755-c955l\" (UID: \"83047914-9b8f-4ea3-9b3e-caea9f8dcaa1\") " pod="openshift-ingress/payload-processing-e2e-api-b-b8fd34-6d55957755-c955l"
Nov 03 09:04:34.853962 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:34.853923    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b8fd34-dockercfg-lflzg\""
Nov 03 09:04:34.853962 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:34.853928    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b8fd34\""
Nov 03 09:04:34.924424 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:34.924397    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8xn7\" (UniqueName: \"kubernetes.io/projected/83047914-9b8f-4ea3-9b3e-caea9f8dcaa1-kube-api-access-p8xn7\") pod \"payload-processing-e2e-api-b-b8fd34-6d55957755-c955l\" (UID: \"83047914-9b8f-4ea3-9b3e-caea9f8dcaa1\") " pod="openshift-ingress/payload-processing-e2e-api-b-b8fd34-6d55957755-c955l"
Nov 03 09:04:34.924609 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:34.924461    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83047914-9b8f-4ea3-9b3e-caea9f8dcaa1-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b8fd34-6d55957755-c955l\" (UID: \"83047914-9b8f-4ea3-9b3e-caea9f8dcaa1\") " pod="openshift-ingress/payload-processing-e2e-api-b-b8fd34-6d55957755-c955l"
Nov 03 09:04:34.925213 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:34.925195    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83047914-9b8f-4ea3-9b3e-caea9f8dcaa1-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b8fd34-6d55957755-c955l\" (UID: \"83047914-9b8f-4ea3-9b3e-caea9f8dcaa1\") " pod="openshift-ingress/payload-processing-e2e-api-b-b8fd34-6d55957755-c955l"
Nov 03 09:04:34.984056 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:34.983990    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8xn7\" (UniqueName: \"kubernetes.io/projected/83047914-9b8f-4ea3-9b3e-caea9f8dcaa1-kube-api-access-p8xn7\") pod \"payload-processing-e2e-api-b-b8fd34-6d55957755-c955l\" (UID: \"83047914-9b8f-4ea3-9b3e-caea9f8dcaa1\") " pod="openshift-ingress/payload-processing-e2e-api-b-b8fd34-6d55957755-c955l"
Nov 03 09:04:35.119420 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:35.119387    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b8fd34-6d55957755-c955l"
Nov 03 09:04:35.497357 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:35.497331    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b8fd34-6d55957755-c955l"]
Nov 03 09:04:35.497472 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:04:35.497422    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83047914_9b8f_4ea3_9b3e_caea9f8dcaa1.slice/crio-1f849eb4970d0f93ba5559dc08254b570e3aa6d6f357900ea58ed9149f69f2e0 WatchSource:0}: Error finding container 1f849eb4970d0f93ba5559dc08254b570e3aa6d6f357900ea58ed9149f69f2e0: Status 404 returned error can't find the container with id 1f849eb4970d0f93ba5559dc08254b570e3aa6d6f357900ea58ed9149f69f2e0
Nov 03 09:04:36.068306 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:36.068275    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 09:04:36.073018 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:36.072988    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b8fd34-6d55957755-c955l" event={"ID":"83047914-9b8f-4ea3-9b3e-caea9f8dcaa1","Type":"ContainerStarted","Data":"a2e9dd25cf9b00a65469ed8c47f899360fdee3bd7562e434c14f9b37c59dee60"}
Nov 03 09:04:36.073146 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:36.073075    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b8fd34-6d55957755-c955l" event={"ID":"83047914-9b8f-4ea3-9b3e-caea9f8dcaa1","Type":"ContainerStarted","Data":"1f849eb4970d0f93ba5559dc08254b570e3aa6d6f357900ea58ed9149f69f2e0"}
Nov 03 09:04:36.073276 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:36.073256    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-b8fd34-6d55957755-c955l"
Nov 03 09:04:36.073623 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:36.073609    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 09:04:36.191122 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:36.191061    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-b8fd34-6d55957755-c955l" podStartSLOduration=2.191043698 podStartE2EDuration="2.191043698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:04:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:04:36.189137144 +0000 UTC m=+1500.458200589" watchObservedRunningTime="2026-11-03 09:04:36.191043698 +0000 UTC m=+1500.460107133"
Nov 03 09:04:38.157421 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:38.157391    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-b0c653f4/e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss"]
Nov 03 09:04:38.161648 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:38.161627    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b0c653f4/e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss"
Nov 03 09:04:38.226171 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:38.226139    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-b0c653f4/e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss"]
Nov 03 09:04:38.268390 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:38.268361    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7213a5c7-1610-4205-8d73-3d096c44c277-dshm\") pod \"e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss\" (UID: \"7213a5c7-1610-4205-8d73-3d096c44c277\") " pod="ai-tenant-e2e-mt-b0c653f4/e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss"
Nov 03 09:04:38.268539 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:38.268401    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7213a5c7-1610-4205-8d73-3d096c44c277-tmp-dir\") pod \"e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss\" (UID: \"7213a5c7-1610-4205-8d73-3d096c44c277\") " pod="ai-tenant-e2e-mt-b0c653f4/e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss"
Nov 03 09:04:38.268539 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:38.268486    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7213a5c7-1610-4205-8d73-3d096c44c277-home\") pod \"e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss\" (UID: \"7213a5c7-1610-4205-8d73-3d096c44c277\") " pod="ai-tenant-e2e-mt-b0c653f4/e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss"
Nov 03 09:04:38.268539 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:38.268527    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8gql\" (UniqueName: \"kubernetes.io/projected/7213a5c7-1610-4205-8d73-3d096c44c277-kube-api-access-j8gql\") pod \"e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss\" (UID: \"7213a5c7-1610-4205-8d73-3d096c44c277\") " pod="ai-tenant-e2e-mt-b0c653f4/e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss"
Nov 03 09:04:38.268680 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:38.268558    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7213a5c7-1610-4205-8d73-3d096c44c277-tls-certs\") pod \"e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss\" (UID: \"7213a5c7-1610-4205-8d73-3d096c44c277\") " pod="ai-tenant-e2e-mt-b0c653f4/e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss"
Nov 03 09:04:38.268680 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:38.268664    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7213a5c7-1610-4205-8d73-3d096c44c277-model-cache\") pod \"e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss\" (UID: \"7213a5c7-1610-4205-8d73-3d096c44c277\") " pod="ai-tenant-e2e-mt-b0c653f4/e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss"
Nov 03 09:04:38.268788 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:38.268758    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-b0c653f4\"/\"kube-root-ca.crt\""
Nov 03 09:04:38.268788 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:38.268759    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-b0c653f4\"/\"openshift-service-ca.crt\""
Nov 03 09:04:38.268788 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:38.268783    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-b0c653f4\"/\"default-dockercfg-lg2tk\""
Nov 03 09:04:38.268911 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:38.268828    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-b0c653f4\"/\"e2e-lifecycle-model-b0c653f4-kserve-self-signed-certs\""
Nov 03 09:04:38.382087 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:38.371557    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7213a5c7-1610-4205-8d73-3d096c44c277-dshm\") pod \"e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss\" (UID: \"7213a5c7-1610-4205-8d73-3d096c44c277\") " pod="ai-tenant-e2e-mt-b0c653f4/e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss"
Nov 03 09:04:38.382087 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:38.372205    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7213a5c7-1610-4205-8d73-3d096c44c277-tmp-dir\") pod \"e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss\" (UID: \"7213a5c7-1610-4205-8d73-3d096c44c277\") " pod="ai-tenant-e2e-mt-b0c653f4/e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss"
Nov 03 09:04:38.382087 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:38.372308    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7213a5c7-1610-4205-8d73-3d096c44c277-home\") pod \"e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss\" (UID: \"7213a5c7-1610-4205-8d73-3d096c44c277\") " pod="ai-tenant-e2e-mt-b0c653f4/e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss"
Nov 03 09:04:38.382087 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:38.376358    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7213a5c7-1610-4205-8d73-3d096c44c277-tmp-dir\") pod \"e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss\" (UID: \"7213a5c7-1610-4205-8d73-3d096c44c277\") " pod="ai-tenant-e2e-mt-b0c653f4/e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss"
Nov 03 09:04:38.387304 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:38.382438    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7213a5c7-1610-4205-8d73-3d096c44c277-home\") pod \"e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss\" (UID: \"7213a5c7-1610-4205-8d73-3d096c44c277\") " pod="ai-tenant-e2e-mt-b0c653f4/e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss"
Nov 03 09:04:38.387304 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:38.386409    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7213a5c7-1610-4205-8d73-3d096c44c277-dshm\") pod \"e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss\" (UID: \"7213a5c7-1610-4205-8d73-3d096c44c277\") " pod="ai-tenant-e2e-mt-b0c653f4/e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss"
Nov 03 09:04:38.387304 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:38.386576    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8gql\" (UniqueName: \"kubernetes.io/projected/7213a5c7-1610-4205-8d73-3d096c44c277-kube-api-access-j8gql\") pod \"e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss\" (UID: \"7213a5c7-1610-4205-8d73-3d096c44c277\") " pod="ai-tenant-e2e-mt-b0c653f4/e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss"
Nov 03 09:04:38.387304 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:38.386623    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7213a5c7-1610-4205-8d73-3d096c44c277-tls-certs\") pod \"e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss\" (UID: \"7213a5c7-1610-4205-8d73-3d096c44c277\") " pod="ai-tenant-e2e-mt-b0c653f4/e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss"
Nov 03 09:04:38.387304 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:38.386719    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7213a5c7-1610-4205-8d73-3d096c44c277-model-cache\") pod \"e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss\" (UID: \"7213a5c7-1610-4205-8d73-3d096c44c277\") " pod="ai-tenant-e2e-mt-b0c653f4/e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss"
Nov 03 09:04:38.387304 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:38.387268    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7213a5c7-1610-4205-8d73-3d096c44c277-model-cache\") pod \"e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss\" (UID: \"7213a5c7-1610-4205-8d73-3d096c44c277\") " pod="ai-tenant-e2e-mt-b0c653f4/e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss"
Nov 03 09:04:38.392133 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:38.392105    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7213a5c7-1610-4205-8d73-3d096c44c277-tls-certs\") pod \"e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss\" (UID: \"7213a5c7-1610-4205-8d73-3d096c44c277\") " pod="ai-tenant-e2e-mt-b0c653f4/e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss"
Nov 03 09:04:38.453834 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:38.453750    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8gql\" (UniqueName: \"kubernetes.io/projected/7213a5c7-1610-4205-8d73-3d096c44c277-kube-api-access-j8gql\") pod \"e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss\" (UID: \"7213a5c7-1610-4205-8d73-3d096c44c277\") " pod="ai-tenant-e2e-mt-b0c653f4/e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss"
Nov 03 09:04:38.478225 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:38.478199    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b0c653f4/e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss"
Nov 03 09:04:38.911707 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:38.911676    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-b0c653f4/e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss"]
Nov 03 09:04:38.912106 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:04:38.912079    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7213a5c7_1610_4205_8d73_3d096c44c277.slice/crio-b2e283007bcb5e73e5ff98af4daab344338818d4cf6b0e7077386fb1444fd8dd WatchSource:0}: Error finding container b2e283007bcb5e73e5ff98af4daab344338818d4cf6b0e7077386fb1444fd8dd: Status 404 returned error can't find the container with id b2e283007bcb5e73e5ff98af4daab344338818d4cf6b0e7077386fb1444fd8dd
Nov 03 09:04:39.093193 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:39.093156    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b0c653f4/e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss" event={"ID":"7213a5c7-1610-4205-8d73-3d096c44c277","Type":"ContainerStarted","Data":"0c62da9e976f8789b9fcfb57a65387671cb3ef5ee47998ee6fd0fa7d814dd8fe"}
Nov 03 09:04:39.093193 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:39.093200    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b0c653f4/e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss" event={"ID":"7213a5c7-1610-4205-8d73-3d096c44c277","Type":"ContainerStarted","Data":"b2e283007bcb5e73e5ff98af4daab344338818d4cf6b0e7077386fb1444fd8dd"}
Nov 03 09:04:39.219045 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:39.218917    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-b0c653f4/e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss" podStartSLOduration=1.218902703 podStartE2EDuration="1.218902703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:04:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:04:39.21539526 +0000 UTC m=+1503.484458713" watchObservedRunningTime="2026-11-03 09:04:39.218902703 +0000 UTC m=+1503.487966151"
Nov 03 09:04:39.479136 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:39.479066    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-b0c653f4/e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss"
Nov 03 09:04:46.937499 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:04:46.937461    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:04:47.082469 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:47.082438    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-b8fd34-6d55957755-c955l"
Nov 03 09:04:47.702546 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:47.702508    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"]
Nov 03 09:04:47.706848 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:47.706824    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"
Nov 03 09:04:47.811619 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:47.811589    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-18f9da-openshift-default-dockercfg-zmkbm\""
Nov 03 09:04:47.815667 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:47.815640    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"]
Nov 03 09:04:47.819506 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:47.819478    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-istio-envoy\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km\" (UID: \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"
Nov 03 09:04:47.819628 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:47.819533    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-credential-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km\" (UID: \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"
Nov 03 09:04:47.819628 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:47.819600    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-istio-token\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km\" (UID: \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"
Nov 03 09:04:47.819736 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:47.819687    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-istiod-ca-cert\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km\" (UID: \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"
Nov 03 09:04:47.819736 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:47.819718    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-workload-certs\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km\" (UID: \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"
Nov 03 09:04:47.819824 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:47.819746    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-istio-data\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km\" (UID: \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"
Nov 03 09:04:47.819824 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:47.819764    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppn75\" (UniqueName: \"kubernetes.io/projected/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-kube-api-access-ppn75\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km\" (UID: \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"
Nov 03 09:04:47.820046 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:47.820024    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-istio-podinfo\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km\" (UID: \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"
Nov 03 09:04:47.820147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:47.820130    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-workload-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km\" (UID: \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"
Nov 03 09:04:47.921645 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:47.921615    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-istio-podinfo\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km\" (UID: \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"
Nov 03 09:04:47.921839 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:47.921678    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-workload-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km\" (UID: \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"
Nov 03 09:04:47.921839 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:47.921736    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-istio-envoy\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km\" (UID: \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"
Nov 03 09:04:47.921839 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:47.921763    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-credential-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km\" (UID: \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"
Nov 03 09:04:47.921839 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:47.921786    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-istio-token\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km\" (UID: \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"
Nov 03 09:04:47.921839 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:47.921833    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-istiod-ca-cert\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km\" (UID: \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"
Nov 03 09:04:47.922134 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:47.921870    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-workload-certs\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km\" (UID: \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"
Nov 03 09:04:47.922134 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:47.921905    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-istio-data\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km\" (UID: \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"
Nov 03 09:04:47.922134 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:47.921932    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppn75\" (UniqueName: \"kubernetes.io/projected/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-kube-api-access-ppn75\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km\" (UID: \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"
Nov 03 09:04:47.922283 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:47.922248    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-workload-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km\" (UID: \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"
Nov 03 09:04:47.922477 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:47.922437    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-istio-data\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km\" (UID: \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"
Nov 03 09:04:47.922559 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:47.922502    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-workload-certs\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km\" (UID: \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"
Nov 03 09:04:47.922880 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:47.922857    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-credential-socket\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km\" (UID: \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"
Nov 03 09:04:47.923008 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:47.922977    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-istiod-ca-cert\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km\" (UID: \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"
Nov 03 09:04:47.924236 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:47.924216    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-istio-envoy\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km\" (UID: \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"
Nov 03 09:04:47.924476 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:47.924459    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-istio-podinfo\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km\" (UID: \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"
Nov 03 09:04:47.949827 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:47.949799    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4417a34e-78fxn"]
Nov 03 09:04:47.954265 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:47.954214    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4417a34e-78fxn"
Nov 03 09:04:47.984013 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:47.983985    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-istio-token\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km\" (UID: \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"
Nov 03 09:04:47.984108 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:47.984008    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppn75\" (UniqueName: \"kubernetes.io/projected/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-kube-api-access-ppn75\") pod \"e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km\" (UID: \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\") " pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"
Nov 03 09:04:48.009105 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:48.009080    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:04:48.009296 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:48.009170    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:04:48.011785 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:48.011759    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4417a34e-78fxn"]
Nov 03 09:04:48.021051 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:48.021026    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"
Nov 03 09:04:48.023723 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:48.023694    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5trc2\" (UniqueName: \"kubernetes.io/projected/7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6-kube-api-access-5trc2\") pod \"maas-api-revoke-sub-4417a34e-78fxn\" (UID: \"7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4417a34e-78fxn"
Nov 03 09:04:48.024008 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:48.023939    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4417a34e-78fxn\" (UID: \"7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4417a34e-78fxn"
Nov 03 09:04:48.024436 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:48.024089    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4417a34e-78fxn\" (UID: \"7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4417a34e-78fxn"
Nov 03 09:04:48.080101 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:48.080078    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:04:48.080232 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:48.080174    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:04:48.125321 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:48.125295    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5trc2\" (UniqueName: \"kubernetes.io/projected/7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6-kube-api-access-5trc2\") pod \"maas-api-revoke-sub-4417a34e-78fxn\" (UID: \"7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4417a34e-78fxn"
Nov 03 09:04:48.125470 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:48.125424    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4417a34e-78fxn\" (UID: \"7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4417a34e-78fxn"
Nov 03 09:04:48.125533 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:48.125485    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4417a34e-78fxn\" (UID: \"7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4417a34e-78fxn"
Nov 03 09:04:48.126069 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:48.126047    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4417a34e-78fxn\" (UID: \"7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4417a34e-78fxn"
Nov 03 09:04:48.127991 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:48.127967    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4417a34e-78fxn\" (UID: \"7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4417a34e-78fxn"
Nov 03 09:04:48.185227 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:48.185195    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5trc2\" (UniqueName: \"kubernetes.io/projected/7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6-kube-api-access-5trc2\") pod \"maas-api-revoke-sub-4417a34e-78fxn\" (UID: \"7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4417a34e-78fxn"
Nov 03 09:04:48.269471 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:48.269397    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4417a34e-78fxn"
Nov 03 09:04:48.451114 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:04:48.451080    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1b72f6f_1e8e_4bb4_a8f6_16eb66cc7ba3.slice/crio-d6f6a3f69251c9edf5943cf68f6de90f802b403ae0204a3fb46c90805f8d7387 WatchSource:0}: Error finding container d6f6a3f69251c9edf5943cf68f6de90f802b403ae0204a3fb46c90805f8d7387: Status 404 returned error can't find the container with id d6f6a3f69251c9edf5943cf68f6de90f802b403ae0204a3fb46c90805f8d7387
Nov 03 09:04:48.451702 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:48.451679    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"]
Nov 03 09:04:48.453450 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:48.453423    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:04:48.453549 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:48.453483    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:04:48.453549 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:48.453511    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:04:48.479043 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:48.479022    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-b0c653f4/e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss"
Nov 03 09:04:48.493339 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:48.493319    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-b0c653f4/e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss"
Nov 03 09:04:48.651288 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:48.651257    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4417a34e-78fxn"]
Nov 03 09:04:48.651596 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:04:48.651571    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b20c997_e1f9_4f94_9c0e_9d03e7d0ecb6.slice/crio-225781b3f17792b48c2cf91916e670eaf94e38e967ab52e950e904c8b079512b WatchSource:0}: Error finding container 225781b3f17792b48c2cf91916e670eaf94e38e967ab52e950e904c8b079512b: Status 404 returned error can't find the container with id 225781b3f17792b48c2cf91916e670eaf94e38e967ab52e950e904c8b079512b
Nov 03 09:04:48.800094 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:48.800065    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92118aae-2jb7c"]
Nov 03 09:04:48.805323 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:48.805304    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92118aae-2jb7c"
Nov 03 09:04:48.834883 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:48.834844    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92118aae-2jb7c\" (UID: \"72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92118aae-2jb7c"
Nov 03 09:04:48.834883 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:48.834876    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg89q\" (UniqueName: \"kubernetes.io/projected/72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087-kube-api-access-pg89q\") pod \"maas-api-revoke-sub-92118aae-2jb7c\" (UID: \"72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92118aae-2jb7c"
Nov 03 09:04:48.835130 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:48.835117    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92118aae-2jb7c\" (UID: \"72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92118aae-2jb7c"
Nov 03 09:04:48.868631 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:48.868602    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92118aae-2jb7c"]
Nov 03 09:04:48.936618 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:48.936531    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92118aae-2jb7c\" (UID: \"72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92118aae-2jb7c"
Nov 03 09:04:48.936618 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:48.936568    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg89q\" (UniqueName: \"kubernetes.io/projected/72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087-kube-api-access-pg89q\") pod \"maas-api-revoke-sub-92118aae-2jb7c\" (UID: \"72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92118aae-2jb7c"
Nov 03 09:04:48.936851 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:48.936738    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92118aae-2jb7c\" (UID: \"72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92118aae-2jb7c"
Nov 03 09:04:48.937339 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:48.937317    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92118aae-2jb7c\" (UID: \"72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92118aae-2jb7c"
Nov 03 09:04:48.939336 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:48.939311    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92118aae-2jb7c\" (UID: \"72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92118aae-2jb7c"
Nov 03 09:04:48.996293 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:48.996267    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg89q\" (UniqueName: \"kubernetes.io/projected/72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087-kube-api-access-pg89q\") pod \"maas-api-revoke-sub-92118aae-2jb7c\" (UID: \"72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92118aae-2jb7c"
Nov 03 09:04:49.127970 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:49.122197    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92118aae-2jb7c"
Nov 03 09:04:49.198761 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:49.198727    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km" event={"ID":"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3","Type":"ContainerStarted","Data":"0b0d3d0082a5ecd43f2e97370b1f448705f8f924998f53d180ceb25eae18d314"}
Nov 03 09:04:49.198856 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:49.198827    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km" event={"ID":"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3","Type":"ContainerStarted","Data":"d6f6a3f69251c9edf5943cf68f6de90f802b403ae0204a3fb46c90805f8d7387"}
Nov 03 09:04:49.200518 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:49.200491    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4417a34e-78fxn" event={"ID":"7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6","Type":"ContainerStarted","Data":"bd84facbf317ca9000e003ba96094b909fa19648493a4d962a2c5a83704f0c69"}
Nov 03 09:04:49.200606 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:49.200541    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4417a34e-78fxn" event={"ID":"7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6","Type":"ContainerStarted","Data":"225781b3f17792b48c2cf91916e670eaf94e38e967ab52e950e904c8b079512b"}
Nov 03 09:04:49.215762 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:49.215722    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-b0c653f4/e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss"
Nov 03 09:04:49.460338 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:49.460230    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4417a34e-78fxn" podStartSLOduration=2.460214143 podStartE2EDuration="2.460214143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:04:49.459209107 +0000 UTC m=+1513.728272544" watchObservedRunningTime="2026-11-03 09:04:49.460214143 +0000 UTC m=+1513.729277577"
Nov 03 09:04:49.460600 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:49.460574    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km" podStartSLOduration=2.460565073 podStartE2EDuration="2.460565073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:04:49.344012048 +0000 UTC m=+1513.613075514" watchObservedRunningTime="2026-11-03 09:04:49.460565073 +0000 UTC m=+1513.729628508"
Nov 03 09:04:49.513610 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:04:49.513585    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b1bb1e_d9d4_4fe7_a2e8_dd0d012c2087.slice/crio-8e0848bc63a4fef75ebab89da8d6e59db76de36e2367e28f0d5667ce2a52cbd1 WatchSource:0}: Error finding container 8e0848bc63a4fef75ebab89da8d6e59db76de36e2367e28f0d5667ce2a52cbd1: Status 404 returned error can't find the container with id 8e0848bc63a4fef75ebab89da8d6e59db76de36e2367e28f0d5667ce2a52cbd1
Nov 03 09:04:49.513727 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:49.513657    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92118aae-2jb7c"]
Nov 03 09:04:50.037684 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:50.037607    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"
Nov 03 09:04:50.042405 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:50.042376    4473 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 09:04:50.042536 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:50.042430    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km" podUID="c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 09:04:50.209280 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:50.209242    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92118aae-2jb7c" event={"ID":"72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087","Type":"ContainerStarted","Data":"23946cddeec9a39b9dbf60111b26051b897ff6db73202e17a7486ba74db82061"}
Nov 03 09:04:50.209482 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:50.209308    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92118aae-2jb7c" event={"ID":"72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087","Type":"ContainerStarted","Data":"8e0848bc63a4fef75ebab89da8d6e59db76de36e2367e28f0d5667ce2a52cbd1"}
Nov 03 09:04:50.211029 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:50.211006    4473 generic.go:356] "Generic (PLEG): container finished" podID="7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6" containerID="bd84facbf317ca9000e003ba96094b909fa19648493a4d962a2c5a83704f0c69" exitCode=0
Nov 03 09:04:50.211157 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:50.211061    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4417a34e-78fxn" event={"ID":"7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6","Type":"ContainerDied","Data":"bd84facbf317ca9000e003ba96094b909fa19648493a4d962a2c5a83704f0c69"}
Nov 03 09:04:50.336022 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:50.335210    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92118aae-2jb7c" podStartSLOduration=2.335191193 podStartE2EDuration="2.335191193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:04:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:04:50.334100565 +0000 UTC m=+1514.603164002" watchObservedRunningTime="2026-11-03 09:04:50.335191193 +0000 UTC m=+1514.604254628"
Nov 03 09:04:51.034388 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:51.034299    4473 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 09:04:51.034806 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:51.034774    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km" podUID="c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 09:04:51.244261 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:51.244222    4473 generic.go:356] "Generic (PLEG): container finished" podID="72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087" containerID="23946cddeec9a39b9dbf60111b26051b897ff6db73202e17a7486ba74db82061" exitCode=0
Nov 03 09:04:51.244808 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:51.244321    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92118aae-2jb7c" event={"ID":"72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087","Type":"ContainerDied","Data":"23946cddeec9a39b9dbf60111b26051b897ff6db73202e17a7486ba74db82061"}
Nov 03 09:04:51.445583 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:51.445549    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4417a34e-78fxn"
Nov 03 09:04:51.491808 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:51.491779    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6-maas-api-xks-ca\") pod \"7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6\" (UID: \"7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6\") "
Nov 03 09:04:51.491969 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:51.491912    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6-kube-api-access-5trc2\" (UniqueName: \"kubernetes.io/projected/7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6-kube-api-access-5trc2\") pod \"7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6\" (UID: \"7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6\") "
Nov 03 09:04:51.492041 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:51.492025    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6-maas-api-openshift-ca\") pod \"7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6\" (UID: \"7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6\") "
Nov 03 09:04:51.492354 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:51.492333    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6-maas-api-openshift-ca" pod "7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6" (UID: "7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:04:51.492570 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:51.492554    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:04:51.494630 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:51.494606    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6-maas-api-xks-ca" pod "7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6" (UID: "7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:04:51.494746 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:51.494628    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6-kube-api-access-5trc2" pod "7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6" (UID: "7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6"). InnerVolumeSpecName "kube-api-access-5trc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:04:51.593808 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:51.593734    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5trc2\" (UniqueName: \"kubernetes.io/projected/7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6-kube-api-access-5trc2\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:04:51.593808 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:51.593763    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:04:52.041016 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:52.037350    4473 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 09:04:52.041016 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:52.037409    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km" podUID="c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 09:04:52.253828 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:52.253750    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4417a34e-78fxn"
Nov 03 09:04:52.253828 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:52.253792    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4417a34e-78fxn" event={"ID":"7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6","Type":"ContainerDied","Data":"225781b3f17792b48c2cf91916e670eaf94e38e967ab52e950e904c8b079512b"}
Nov 03 09:04:52.254315 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:52.253835    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="225781b3f17792b48c2cf91916e670eaf94e38e967ab52e950e904c8b079512b"
Nov 03 09:04:52.446750 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:52.446717    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92118aae-2jb7c"
Nov 03 09:04:52.503590 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:52.503563    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087-kube-api-access-pg89q\" (UniqueName: \"kubernetes.io/projected/72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087-kube-api-access-pg89q\") pod \"72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087\" (UID: \"72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087\") "
Nov 03 09:04:52.503725 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:52.503603    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087-maas-api-openshift-ca\") pod \"72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087\" (UID: \"72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087\") "
Nov 03 09:04:52.503725 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:52.503659    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087-maas-api-xks-ca\") pod \"72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087\" (UID: \"72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087\") "
Nov 03 09:04:52.504024 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:52.503972    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087-maas-api-openshift-ca" pod "72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087" (UID: "72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:04:52.504197 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:52.504181    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:04:52.505823 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:52.505803    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087-kube-api-access-pg89q" pod "72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087" (UID: "72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087"). InnerVolumeSpecName "kube-api-access-pg89q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:04:52.505877 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:52.505861    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087-maas-api-xks-ca" pod "72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087" (UID: "72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:04:52.605217 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:52.605187    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg89q\" (UniqueName: \"kubernetes.io/projected/72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087-kube-api-access-pg89q\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:04:52.605217 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:52.605214    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:04:53.041070 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:53.041038    4473 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 09:04:53.041250 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:53.041087    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km" podUID="c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 09:04:53.261823 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:53.261796    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92118aae-2jb7c"
Nov 03 09:04:53.262198 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:53.261840    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92118aae-2jb7c" event={"ID":"72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087","Type":"ContainerDied","Data":"8e0848bc63a4fef75ebab89da8d6e59db76de36e2367e28f0d5667ce2a52cbd1"}
Nov 03 09:04:53.262198 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:53.261891    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e0848bc63a4fef75ebab89da8d6e59db76de36e2367e28f0d5667ce2a52cbd1"
Nov 03 09:04:54.026971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:54.023309    4473 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 09:04:54.026971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:54.023366    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km" podUID="c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 09:04:55.021463 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:55.021426    4473 patch_prober.go:29] interesting pod/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Nov 03 09:04:55.021855 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:55.021498    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km" podUID="c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Nov 03 09:04:56.025894 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:56.025863    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"
Nov 03 09:04:56.026376 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:56.026200    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"
Nov 03 09:04:56.027016 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:56.027000    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"
Nov 03 09:04:57.077764 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:04:57.077732    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:05:16.730982 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:16.730098    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-whmmw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 09:05:16.730982 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:16.730185    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw" podUID="14dd6c95-a250-41c5-87c7-ae741f96d01a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:05:16.893971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:16.890048    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx" podUID="8e84a474-def6-41dc-8516-5e5ccf42ff46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:05:16.920581 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:16.920532    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn" podUID="035b0461-2dfa-42f4-8ea7-2d1911cddca9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 09:05:17.106434 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:05:17.104119    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:05:25.600190 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:25.600159    4473 generic.go:356] "Generic (PLEG): container finished" podID="625c61e9-f360-4f60-9b21-885c4696aba3" containerID="8e2707fcf56521b5ddf62f7183fd97098e64a10cd344cb52fecbaf9ac901a00e" exitCode=0
Nov 03 09:05:25.600703 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:25.600229    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68" event={"ID":"625c61e9-f360-4f60-9b21-885c4696aba3","Type":"ContainerDied","Data":"8e2707fcf56521b5ddf62f7183fd97098e64a10cd344cb52fecbaf9ac901a00e"}
Nov 03 09:05:25.600703 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:25.600656    4473 scope.go:122] "RemoveContainer" containerID="8e2707fcf56521b5ddf62f7183fd97098e64a10cd344cb52fecbaf9ac901a00e"
Nov 03 09:05:25.602212 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:25.602189    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:05:25.602280 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:25.602254    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:05:25.602316 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:25.602286    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:05:25.838988 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:25.838931    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:05:26.611224 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:26.611187    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68" event={"ID":"625c61e9-f360-4f60-9b21-885c4696aba3","Type":"ContainerStarted","Data":"90d3b4e38efc4635aaa4742962ac9f70cd966ca053357c1cf00b6a4949c8f2e4"}
Nov 03 09:05:27.538970 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:27.538272    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:05:27.541165 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:27.541112    4473 patch_prober.go:29] interesting pod/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Nov 03 09:05:27.541310 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:27.541195    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68" podUID="625c61e9-f360-4f60-9b21-885c4696aba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Nov 03 09:05:28.537970 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:28.536334    4473 patch_prober.go:29] interesting pod/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Nov 03 09:05:28.537970 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:28.536394    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68" podUID="625c61e9-f360-4f60-9b21-885c4696aba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Nov 03 09:05:29.536970 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:29.535972    4473 patch_prober.go:29] interesting pod/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Nov 03 09:05:29.536970 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:29.536031    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68" podUID="625c61e9-f360-4f60-9b21-885c4696aba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Nov 03 09:05:29.701503 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:29.701468    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1993359e-xfx7p"]
Nov 03 09:05:29.702043 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:29.702026    4473 cpu_manager.go:414] "containerMap: removing container" podUID="72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087" containerName="revoke-keys"
Nov 03 09:05:29.702101 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:29.702044    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087" containerName="revoke-keys"
Nov 03 09:05:29.702101 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:29.702083    4473 cpu_manager.go:414] "containerMap: removing container" podUID="7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6" containerName="revoke-keys"
Nov 03 09:05:29.702101 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:29.702088    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6" containerName="revoke-keys"
Nov 03 09:05:29.702202 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:29.702183    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6" containerName="revoke-keys"
Nov 03 09:05:29.702202 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:29.702191    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087" containerName="revoke-keys"
Nov 03 09:05:29.706147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:29.706131    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1993359e-xfx7p"
Nov 03 09:05:29.775740 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:29.775571    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1993359e-xfx7p"]
Nov 03 09:05:29.795303 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:29.795221    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9347af0f-ab7f-4949-8642-a92f82388f44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1993359e-xfx7p\" (UID: \"9347af0f-ab7f-4949-8642-a92f82388f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1993359e-xfx7p"
Nov 03 09:05:29.795540 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:29.795350    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkt95\" (UniqueName: \"kubernetes.io/projected/9347af0f-ab7f-4949-8642-a92f82388f44-kube-api-access-rkt95\") pod \"maas-api-revoke-sub-1993359e-xfx7p\" (UID: \"9347af0f-ab7f-4949-8642-a92f82388f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1993359e-xfx7p"
Nov 03 09:05:29.795603 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:29.795549    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9347af0f-ab7f-4949-8642-a92f82388f44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1993359e-xfx7p\" (UID: \"9347af0f-ab7f-4949-8642-a92f82388f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1993359e-xfx7p"
Nov 03 09:05:29.841607 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:29.826178    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:05:29.841607 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:29.826252    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:05:29.841607 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:29.826301    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:05:29.841607 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:29.826361    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:05:29.896323 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:29.896291    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkt95\" (UniqueName: \"kubernetes.io/projected/9347af0f-ab7f-4949-8642-a92f82388f44-kube-api-access-rkt95\") pod \"maas-api-revoke-sub-1993359e-xfx7p\" (UID: \"9347af0f-ab7f-4949-8642-a92f82388f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1993359e-xfx7p"
Nov 03 09:05:29.896916 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:29.896535    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9347af0f-ab7f-4949-8642-a92f82388f44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1993359e-xfx7p\" (UID: \"9347af0f-ab7f-4949-8642-a92f82388f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1993359e-xfx7p"
Nov 03 09:05:29.896916 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:29.896739    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9347af0f-ab7f-4949-8642-a92f82388f44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1993359e-xfx7p\" (UID: \"9347af0f-ab7f-4949-8642-a92f82388f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1993359e-xfx7p"
Nov 03 09:05:29.897390 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:29.897365    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9347af0f-ab7f-4949-8642-a92f82388f44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1993359e-xfx7p\" (UID: \"9347af0f-ab7f-4949-8642-a92f82388f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1993359e-xfx7p"
Nov 03 09:05:29.899291 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:29.899272    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9347af0f-ab7f-4949-8642-a92f82388f44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1993359e-xfx7p\" (UID: \"9347af0f-ab7f-4949-8642-a92f82388f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1993359e-xfx7p"
Nov 03 09:05:29.955561 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:29.955532    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkt95\" (UniqueName: \"kubernetes.io/projected/9347af0f-ab7f-4949-8642-a92f82388f44-kube-api-access-rkt95\") pod \"maas-api-revoke-sub-1993359e-xfx7p\" (UID: \"9347af0f-ab7f-4949-8642-a92f82388f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1993359e-xfx7p"
Nov 03 09:05:30.041229 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:30.041190    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1993359e-xfx7p"
Nov 03 09:05:30.440151 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:30.437095    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1993359e-xfx7p"]
Nov 03 09:05:30.540972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:30.534519    4473 patch_prober.go:29] interesting pod/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Nov 03 09:05:30.540972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:30.534575    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68" podUID="625c61e9-f360-4f60-9b21-885c4696aba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Nov 03 09:05:30.660793 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:30.660764    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1993359e-xfx7p" event={"ID":"9347af0f-ab7f-4949-8642-a92f82388f44","Type":"ContainerStarted","Data":"9855df2432bdc2bbeb3882ea44e995b27fb8e5239421d68cda01ca80ca1bd2bb"}
Nov 03 09:05:31.531491 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:31.531450    4473 patch_prober.go:29] interesting pod/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Nov 03 09:05:31.531908 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:31.531519    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68" podUID="625c61e9-f360-4f60-9b21-885c4696aba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Nov 03 09:05:31.670707 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:31.670677    4473 generic.go:356] "Generic (PLEG): container finished" podID="9347af0f-ab7f-4949-8642-a92f82388f44" containerID="f7faea6109fb78f5f36adccfeae4b736403a96444f93f888ebfc36a282b2c7c3" exitCode=0
Nov 03 09:05:31.670841 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:31.670764    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1993359e-xfx7p" event={"ID":"9347af0f-ab7f-4949-8642-a92f82388f44","Type":"ContainerDied","Data":"f7faea6109fb78f5f36adccfeae4b736403a96444f93f888ebfc36a282b2c7c3"}
Nov 03 09:05:32.055552 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:32.055523    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:05:32.536764 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:32.536726    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:05:32.537246 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:32.537161    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:05:32.537756 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:32.537740    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:05:33.011686 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:33.011658    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1993359e-xfx7p"
Nov 03 09:05:33.103490 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:33.103459    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9347af0f-ab7f-4949-8642-a92f82388f44-kube-api-access-rkt95\" (UniqueName: \"kubernetes.io/projected/9347af0f-ab7f-4949-8642-a92f82388f44-kube-api-access-rkt95\") pod \"9347af0f-ab7f-4949-8642-a92f82388f44\" (UID: \"9347af0f-ab7f-4949-8642-a92f82388f44\") "
Nov 03 09:05:33.103705 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:33.103686    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9347af0f-ab7f-4949-8642-a92f82388f44-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9347af0f-ab7f-4949-8642-a92f82388f44-maas-api-openshift-ca\") pod \"9347af0f-ab7f-4949-8642-a92f82388f44\" (UID: \"9347af0f-ab7f-4949-8642-a92f82388f44\") "
Nov 03 09:05:33.103773 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:33.103753    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9347af0f-ab7f-4949-8642-a92f82388f44-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9347af0f-ab7f-4949-8642-a92f82388f44-maas-api-xks-ca\") pod \"9347af0f-ab7f-4949-8642-a92f82388f44\" (UID: \"9347af0f-ab7f-4949-8642-a92f82388f44\") "
Nov 03 09:05:33.104121 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:33.104095    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9347af0f-ab7f-4949-8642-a92f82388f44-maas-api-openshift-ca" pod "9347af0f-ab7f-4949-8642-a92f82388f44" (UID: "9347af0f-ab7f-4949-8642-a92f82388f44"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:05:33.104423 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:33.104405    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9347af0f-ab7f-4949-8642-a92f82388f44-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:05:33.105774 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:33.105756    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9347af0f-ab7f-4949-8642-a92f82388f44-kube-api-access-rkt95" pod "9347af0f-ab7f-4949-8642-a92f82388f44" (UID: "9347af0f-ab7f-4949-8642-a92f82388f44"). InnerVolumeSpecName "kube-api-access-rkt95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:05:33.105905 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:33.105884    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9347af0f-ab7f-4949-8642-a92f82388f44-maas-api-xks-ca" pod "9347af0f-ab7f-4949-8642-a92f82388f44" (UID: "9347af0f-ab7f-4949-8642-a92f82388f44"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:05:33.205866 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:33.205790    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9347af0f-ab7f-4949-8642-a92f82388f44-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:05:33.205866 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:33.205816    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkt95\" (UniqueName: \"kubernetes.io/projected/9347af0f-ab7f-4949-8642-a92f82388f44-kube-api-access-rkt95\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:05:33.715969 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:33.715075    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1993359e-xfx7p"
Nov 03 09:05:33.715969 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:33.715098    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1993359e-xfx7p" event={"ID":"9347af0f-ab7f-4949-8642-a92f82388f44","Type":"ContainerDied","Data":"9855df2432bdc2bbeb3882ea44e995b27fb8e5239421d68cda01ca80ca1bd2bb"}
Nov 03 09:05:33.715969 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:33.715171    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9855df2432bdc2bbeb3882ea44e995b27fb8e5239421d68cda01ca80ca1bd2bb"
Nov 03 09:05:34.056010 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:34.055887    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70c3db8c-r64gq"]
Nov 03 09:05:34.068388 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:34.068354    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70c3db8c-r64gq"]
Nov 03 09:05:35.870689 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:35.870654    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9abf6bd6-23a7-4398-92ab-27c1d16ac7f9" path="/var/lib/kubelet/pods/9abf6bd6-23a7-4398-92ab-27c1d16ac7f9/volumes"
Nov 03 09:05:36.313672 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:36.313591    4473 scope.go:122] "RemoveContainer" containerID="5f5560fd4725dab1b9175d43ae43590103745ab1bc14673883828f02a66fde76"
Nov 03 09:05:42.807051 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:42.807024    4473 generic.go:356] "Generic (PLEG): container finished" podID="6e740caf-68ae-4528-8965-a1085396deca" containerID="88b7efd12000a2b1277ba153152f5a01be2ab91999f6116d182a19e1545b52c2" exitCode=0
Nov 03 09:05:42.807414 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:42.807098    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr" event={"ID":"6e740caf-68ae-4528-8965-a1085396deca","Type":"ContainerDied","Data":"88b7efd12000a2b1277ba153152f5a01be2ab91999f6116d182a19e1545b52c2"}
Nov 03 09:05:42.807545 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:42.807532    4473 scope.go:122] "RemoveContainer" containerID="88b7efd12000a2b1277ba153152f5a01be2ab91999f6116d182a19e1545b52c2"
Nov 03 09:05:42.808913 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:42.808889    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:05:42.808996 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:42.808982    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:05:42.809035 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:42.809018    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:05:43.815660 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:43.815633    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr" event={"ID":"6e740caf-68ae-4528-8965-a1085396deca","Type":"ContainerStarted","Data":"57e12e2404ff3e78f53433b36e64c37cba27cc506de5459f2697659cda902595"}
Nov 03 09:05:43.992763 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:43.992730    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"
Nov 03 09:05:44.270421 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:44.270386    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"
Nov 03 09:05:44.271714 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:44.271693    4473 patch_prober.go:29] interesting pod/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 09:05:44.271775 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:44.271755    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr" podUID="6e740caf-68ae-4528-8965-a1085396deca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 09:05:45.269793 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:45.269765    4473 patch_prober.go:29] interesting pod/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 09:05:45.270193 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:45.269813    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr" podUID="6e740caf-68ae-4528-8965-a1085396deca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 09:05:46.270555 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:46.270520    4473 patch_prober.go:29] interesting pod/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 09:05:46.271000 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:46.270571    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr" podUID="6e740caf-68ae-4528-8965-a1085396deca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 09:05:46.963175 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:05:46.963098    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:05:47.054766 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:47.054727    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49033a88-gst8z"]
Nov 03 09:05:47.105120 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:47.105092    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49033a88-gst8z"]
Nov 03 09:05:47.270306 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:47.270226    4473 patch_prober.go:29] interesting pod/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 09:05:47.270306 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:47.270278    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr" podUID="6e740caf-68ae-4528-8965-a1085396deca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 09:05:47.871006 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:47.870973    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74" path="/var/lib/kubelet/pods/ad684a9f-e9eb-4c0c-b3e3-5471b3ebbd74/volumes"
Nov 03 09:05:48.270320 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:48.270244    4473 patch_prober.go:29] interesting pod/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 03 09:05:48.270320 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:48.270299    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr" podUID="6e740caf-68ae-4528-8965-a1085396deca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 03 09:05:49.274467 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:49.274439    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"
Nov 03 09:05:49.275555 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:49.275534    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"
Nov 03 09:05:55.473277 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:55.473243    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"]
Nov 03 09:05:55.473787 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:55.473772    4473 cpu_manager.go:414] "containerMap: removing container" podUID="9347af0f-ab7f-4949-8642-a92f82388f44" containerName="revoke-keys"
Nov 03 09:05:55.473831 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:55.473788    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9347af0f-ab7f-4949-8642-a92f82388f44" containerName="revoke-keys"
Nov 03 09:05:55.473911 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:55.473902    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="9347af0f-ab7f-4949-8642-a92f82388f44" containerName="revoke-keys"
Nov 03 09:05:55.479972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:55.479923    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"
Nov 03 09:05:55.579514 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:55.579467    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-credential-socket\") pod \"e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w\" (UID: \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\") " pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"
Nov 03 09:05:55.579861 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:55.579833    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-istio-envoy\") pod \"e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w\" (UID: \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\") " pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"
Nov 03 09:05:55.580204 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:55.580183    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-workload-socket\") pod \"e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w\" (UID: \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\") " pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"
Nov 03 09:05:55.588495 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:55.588470    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-istio-podinfo\") pod \"e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w\" (UID: \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\") " pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"
Nov 03 09:05:55.588821 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:55.588799    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-istiod-ca-cert\") pod \"e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w\" (UID: \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\") " pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"
Nov 03 09:05:55.589030 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:55.589010    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkdfj\" (UniqueName: \"kubernetes.io/projected/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-kube-api-access-wkdfj\") pod \"e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w\" (UID: \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\") " pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"
Nov 03 09:05:55.589211 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:55.589184    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-istio-data\") pod \"e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w\" (UID: \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\") " pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"
Nov 03 09:05:55.594971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:55.589416    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-workload-certs\") pod \"e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w\" (UID: \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\") " pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"
Nov 03 09:05:55.600974 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:55.595278    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-istio-token\") pod \"e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w\" (UID: \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\") " pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"
Nov 03 09:05:55.633036 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:55.595049    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-eb9cd2-openshift-default-dockercfg-pjn7v\""
Nov 03 09:05:55.633336 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:55.603640    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"]
Nov 03 09:05:55.742750 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:55.742662    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-workload-certs\") pod \"e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w\" (UID: \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\") " pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"
Nov 03 09:05:55.743141 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:55.743105    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-istio-token\") pod \"e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w\" (UID: \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\") " pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"
Nov 03 09:05:55.743301 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:55.743289    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-credential-socket\") pod \"e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w\" (UID: \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\") " pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"
Nov 03 09:05:55.743374 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:55.743324    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-workload-certs\") pod \"e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w\" (UID: \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\") " pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"
Nov 03 09:05:55.743644 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:55.743627    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-credential-socket\") pod \"e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w\" (UID: \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\") " pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"
Nov 03 09:05:55.743725 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:55.743635    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-istio-envoy\") pod \"e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w\" (UID: \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\") " pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"
Nov 03 09:05:55.743802 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:55.743791    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-workload-socket\") pod \"e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w\" (UID: \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\") " pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"
Nov 03 09:05:55.743894 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:55.743886    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-istio-podinfo\") pod \"e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w\" (UID: \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\") " pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"
Nov 03 09:05:55.746509 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:55.746487    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-istiod-ca-cert\") pod \"e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w\" (UID: \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\") " pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"
Nov 03 09:05:55.746720 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:55.746703    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkdfj\" (UniqueName: \"kubernetes.io/projected/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-kube-api-access-wkdfj\") pod \"e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w\" (UID: \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\") " pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"
Nov 03 09:05:55.747060 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:55.747040    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-istio-data\") pod \"e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w\" (UID: \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\") " pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"
Nov 03 09:05:55.747606 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:55.747585    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-istio-data\") pod \"e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w\" (UID: \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\") " pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"
Nov 03 09:05:55.747738 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:55.744723    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-workload-socket\") pod \"e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w\" (UID: \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\") " pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"
Nov 03 09:05:55.748504 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:55.748483    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-istiod-ca-cert\") pod \"e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w\" (UID: \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\") " pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"
Nov 03 09:05:55.748651 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:55.748525    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-istio-envoy\") pod \"e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w\" (UID: \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\") " pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"
Nov 03 09:05:55.750989 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:55.749478    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-istio-podinfo\") pod \"e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w\" (UID: \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\") " pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"
Nov 03 09:05:55.802831 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:55.802800    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-istio-token\") pod \"e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w\" (UID: \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\") " pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"
Nov 03 09:05:55.807391 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:55.807366    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkdfj\" (UniqueName: \"kubernetes.io/projected/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-kube-api-access-wkdfj\") pod \"e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w\" (UID: \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\") " pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"
Nov 03 09:05:55.837654 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:55.837626    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"
Nov 03 09:05:56.428813 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:05:56.428777    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4cf07f_a8e3_4cda_96d9_7e0b8f331bc5.slice/crio-da768deed4b0e054e6c49e7b7d3810f4371c46306db7b316cd3cf48072d79a5b WatchSource:0}: Error finding container da768deed4b0e054e6c49e7b7d3810f4371c46306db7b316cd3cf48072d79a5b: Status 404 returned error can't find the container with id da768deed4b0e054e6c49e7b7d3810f4371c46306db7b316cd3cf48072d79a5b
Nov 03 09:05:56.431158 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:56.431133    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"]
Nov 03 09:05:56.431474 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:56.431444    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:05:56.431546 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:56.431528    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:05:56.431614 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:56.431595    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:05:57.004995 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:57.004054    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w" event={"ID":"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5","Type":"ContainerStarted","Data":"a3d29a7c4944cc360ce95640deabd3a70173b062a18f843efad97f5e894c3b01"}
Nov 03 09:05:57.004995 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:57.004122    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w" event={"ID":"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5","Type":"ContainerStarted","Data":"da768deed4b0e054e6c49e7b7d3810f4371c46306db7b316cd3cf48072d79a5b"}
Nov 03 09:05:57.177092 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:57.177043    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w" podStartSLOduration=2.177030136 podStartE2EDuration="2.177030136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:05:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:05:57.174744081 +0000 UTC m=+1581.443807539" watchObservedRunningTime="2026-11-03 09:05:57.177030136 +0000 UTC m=+1581.446093569"
Nov 03 09:05:57.838289 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:57.838255    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"
Nov 03 09:05:57.843840 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:57.843799    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"
Nov 03 09:05:58.010477 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:58.010446    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"
Nov 03 09:05:58.011383 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:58.011367    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"
Nov 03 09:05:58.022920 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:05:58.022894    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:06:01.057895 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:01.057864    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d2fc1b5-mbssl"]
Nov 03 09:06:01.108596 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:01.108564    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d2fc1b5-mbssl"]
Nov 03 09:06:01.870229 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:01.870194    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="346fcfc5-c545-4443-aae5-f3945c68b8fe" path="/var/lib/kubelet/pods/346fcfc5-c545-4443-aae5-f3945c68b8fe/volumes"
Nov 03 09:06:12.056246 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:12.056213    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3db44a7d-dpkkq"]
Nov 03 09:06:12.105453 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:12.105425    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3db44a7d-dpkkq"]
Nov 03 09:06:13.871055 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:13.871023    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db1f2d7e-573d-48b5-aa1c-c89f8224797c" path="/var/lib/kubelet/pods/db1f2d7e-573d-48b5-aa1c-c89f8224797c/volumes"
Nov 03 09:06:14.642703 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:14.642670    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7378-dceffd"]
Nov 03 09:06:14.648017 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:14.647998    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7378-dceffd"
Nov 03 09:06:14.703000 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:14.702969    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7378-dceffd"]
Nov 03 09:06:14.737139 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:14.737101    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcprx\" (UniqueName: \"kubernetes.io/projected/fc4b1919-78b5-46e8-b7ff-2f4a3a048ff3-kube-api-access-xcprx\") pod \"test-curl-7378-dceffd\" (UID: \"fc4b1919-78b5-46e8-b7ff-2f4a3a048ff3\") " pod="opendatahub/test-curl-7378-dceffd"
Nov 03 09:06:14.746718 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:14.746693    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-77bkh\""
Nov 03 09:06:14.838785 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:14.838756    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcprx\" (UniqueName: \"kubernetes.io/projected/fc4b1919-78b5-46e8-b7ff-2f4a3a048ff3-kube-api-access-xcprx\") pod \"test-curl-7378-dceffd\" (UID: \"fc4b1919-78b5-46e8-b7ff-2f4a3a048ff3\") " pod="opendatahub/test-curl-7378-dceffd"
Nov 03 09:06:14.898192 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:14.898116    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcprx\" (UniqueName: \"kubernetes.io/projected/fc4b1919-78b5-46e8-b7ff-2f4a3a048ff3-kube-api-access-xcprx\") pod \"test-curl-7378-dceffd\" (UID: \"fc4b1919-78b5-46e8-b7ff-2f4a3a048ff3\") " pod="opendatahub/test-curl-7378-dceffd"
Nov 03 09:06:14.961062 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:14.961038    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7378-dceffd"
Nov 03 09:06:15.342934 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:15.342905    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7378-dceffd"]
Nov 03 09:06:15.343149 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:06:15.343128    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc4b1919_78b5_46e8_b7ff_2f4a3a048ff3.slice/crio-51b403c58197edc9029a959aefe6075a708ac17992549c7d94b4203d19bcefb7 WatchSource:0}: Error finding container 51b403c58197edc9029a959aefe6075a708ac17992549c7d94b4203d19bcefb7: Status 404 returned error can't find the container with id 51b403c58197edc9029a959aefe6075a708ac17992549c7d94b4203d19bcefb7
Nov 03 09:06:15.345165 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:15.345141    4473 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 09:06:16.134215 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:16.134181    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7378-dceffd" event={"ID":"fc4b1919-78b5-46e8-b7ff-2f4a3a048ff3","Type":"ContainerStarted","Data":"51b403c58197edc9029a959aefe6075a708ac17992549c7d94b4203d19bcefb7"}
Nov 03 09:06:16.949175 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:06:16.949100    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:06:19.053523 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:19.053447    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cbe5945-l5pxb"]
Nov 03 09:06:19.103462 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:19.103437    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cbe5945-l5pxb"]
Nov 03 09:06:19.161902 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:19.161861    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7378-dceffd" event={"ID":"fc4b1919-78b5-46e8-b7ff-2f4a3a048ff3","Type":"ContainerStarted","Data":"df6c4bc4afe5356c62d0c8c29b774879e91acdb8d73da262559c00519b8e5ce7"}
Nov 03 09:06:19.280040 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:19.279989    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7378-dceffd" podStartSLOduration=1.8771552580000002 podStartE2EDuration="5.279975921s" totalImagesPullingTime="3.402820663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:06:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:06:19.276177554 +0000 UTC m=+1603.545240999" watchObservedRunningTime="2026-11-03 09:06:19.279975921 +0000 UTC m=+1603.549039355"
Nov 03 09:06:19.901620 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:19.901075    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d" path="/var/lib/kubelet/pods/bfd4a6a8-61ed-44ca-acaf-6cbe6455bb4d/volumes"
Nov 03 09:06:20.754304 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:20.754267    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7378-dceffd"]
Nov 03 09:06:20.762691 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:20.762663    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7378-dceffd"]
Nov 03 09:06:21.069186 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:21.069113    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0be8722-6lsc2"]
Nov 03 09:06:21.073323 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:21.073300    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0be8722-6lsc2"]
Nov 03 09:06:21.185639 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:21.185591    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7378-dceffd" podUID="fc4b1919-78b5-46e8-b7ff-2f4a3a048ff3" containerName="test-curl-7378-dceffd" containerID="cri-o://df6c4bc4afe5356c62d0c8c29b774879e91acdb8d73da262559c00519b8e5ce7" gracePeriod=2
Nov 03 09:06:21.237027 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:06:21.236990    4473 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7378-dceffd\" is forbidden: User \"system:node:ip-10-0-139-73.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-73.us-west-1.compute.internal' and this object" podUID="fc4b1919-78b5-46e8-b7ff-2f4a3a048ff3" pod="opendatahub/test-curl-7378-dceffd"
Nov 03 09:06:21.731967 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:21.731923    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b0c653f4-fp44f"]
Nov 03 09:06:21.732579 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:21.732560    4473 cpu_manager.go:414] "containerMap: removing container" podUID="fc4b1919-78b5-46e8-b7ff-2f4a3a048ff3" containerName="test-curl-7378-dceffd"
Nov 03 09:06:21.732579 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:21.732579    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc4b1919-78b5-46e8-b7ff-2f4a3a048ff3" containerName="test-curl-7378-dceffd"
Nov 03 09:06:21.732714 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:21.732657    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc4b1919-78b5-46e8-b7ff-2f4a3a048ff3" containerName="test-curl-7378-dceffd"
Nov 03 09:06:21.736345 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:21.736330    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b0c653f4-fp44f"
Nov 03 09:06:21.791172 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:21.791149    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:06:21.794829 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:21.794791    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b0c653f4-fp44f"]
Nov 03 09:06:21.840002 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:21.839977    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:06:21.840002 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:21.839986    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:06:21.840154 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:21.839987    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:06:21.870184 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:21.870155    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e8747ce-e051-435a-a5d9-bdd4e3991391" path="/var/lib/kubelet/pods/7e8747ce-e051-435a-a5d9-bdd4e3991391/volumes"
Nov 03 09:06:21.877304 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:21.877281    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjr4x\" (UniqueName: \"kubernetes.io/projected/fa3aefbe-61d6-476e-9a87-d86fec01ecaf-kube-api-access-vjr4x\") pod \"maas-api-revoke-keys-e2e-mt-b0c653f4-fp44f\" (UID: \"fa3aefbe-61d6-476e-9a87-d86fec01ecaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b0c653f4-fp44f"
Nov 03 09:06:21.877434 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:21.877345    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa3aefbe-61d6-476e-9a87-d86fec01ecaf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-b0c653f4-fp44f\" (UID: \"fa3aefbe-61d6-476e-9a87-d86fec01ecaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b0c653f4-fp44f"
Nov 03 09:06:21.877522 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:21.877506    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa3aefbe-61d6-476e-9a87-d86fec01ecaf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-b0c653f4-fp44f\" (UID: \"fa3aefbe-61d6-476e-9a87-d86fec01ecaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b0c653f4-fp44f"
Nov 03 09:06:21.902882 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:06:21.902851    4473 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7378-dceffd\" is forbidden: User \"system:node:ip-10-0-139-73.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-73.us-west-1.compute.internal' and this object" podUID="fc4b1919-78b5-46e8-b7ff-2f4a3a048ff3" pod="opendatahub/test-curl-7378-dceffd"
Nov 03 09:06:21.978551 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:21.978517    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjr4x\" (UniqueName: \"kubernetes.io/projected/fa3aefbe-61d6-476e-9a87-d86fec01ecaf-kube-api-access-vjr4x\") pod \"maas-api-revoke-keys-e2e-mt-b0c653f4-fp44f\" (UID: \"fa3aefbe-61d6-476e-9a87-d86fec01ecaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b0c653f4-fp44f"
Nov 03 09:06:21.978714 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:21.978602    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa3aefbe-61d6-476e-9a87-d86fec01ecaf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-b0c653f4-fp44f\" (UID: \"fa3aefbe-61d6-476e-9a87-d86fec01ecaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b0c653f4-fp44f"
Nov 03 09:06:21.978714 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:21.978681    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa3aefbe-61d6-476e-9a87-d86fec01ecaf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-b0c653f4-fp44f\" (UID: \"fa3aefbe-61d6-476e-9a87-d86fec01ecaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b0c653f4-fp44f"
Nov 03 09:06:21.979250 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:21.979230    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa3aefbe-61d6-476e-9a87-d86fec01ecaf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-b0c653f4-fp44f\" (UID: \"fa3aefbe-61d6-476e-9a87-d86fec01ecaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b0c653f4-fp44f"
Nov 03 09:06:21.981115 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:21.981097    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa3aefbe-61d6-476e-9a87-d86fec01ecaf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-b0c653f4-fp44f\" (UID: \"fa3aefbe-61d6-476e-9a87-d86fec01ecaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b0c653f4-fp44f"
Nov 03 09:06:22.041192 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:22.041125    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjr4x\" (UniqueName: \"kubernetes.io/projected/fa3aefbe-61d6-476e-9a87-d86fec01ecaf-kube-api-access-vjr4x\") pod \"maas-api-revoke-keys-e2e-mt-b0c653f4-fp44f\" (UID: \"fa3aefbe-61d6-476e-9a87-d86fec01ecaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b0c653f4-fp44f"
Nov 03 09:06:22.049997 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:22.049974    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b0c653f4-fp44f"
Nov 03 09:06:22.430559 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:22.430511    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b0c653f4-fp44f"]
Nov 03 09:06:22.430780 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:06:22.430754    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa3aefbe_61d6_476e_9a87_d86fec01ecaf.slice/crio-68182f4768ce7fbdbabc18bc5d4283b43211935871a79601215b356d9395930a WatchSource:0}: Error finding container 68182f4768ce7fbdbabc18bc5d4283b43211935871a79601215b356d9395930a: Status 404 returned error can't find the container with id 68182f4768ce7fbdbabc18bc5d4283b43211935871a79601215b356d9395930a
Nov 03 09:06:23.201621 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:23.201585    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b0c653f4-fp44f" event={"ID":"fa3aefbe-61d6-476e-9a87-d86fec01ecaf","Type":"ContainerStarted","Data":"86a3a6858ed6adc436573c725e9dcf611d641d21443d2cda69cd7663aa44cde5"}
Nov 03 09:06:23.202081 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:23.201684    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b0c653f4-fp44f" event={"ID":"fa3aefbe-61d6-476e-9a87-d86fec01ecaf","Type":"ContainerStarted","Data":"68182f4768ce7fbdbabc18bc5d4283b43211935871a79601215b356d9395930a"}
Nov 03 09:06:23.318833 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:23.318775    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b0c653f4-fp44f" podStartSLOduration=2.318759143 podStartE2EDuration="2.318759143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:06:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:06:23.317006287 +0000 UTC m=+1607.586069744" watchObservedRunningTime="2026-11-03 09:06:23.318759143 +0000 UTC m=+1607.587822576"
Nov 03 09:06:23.484414 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:23.484391    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7378-dceffd"
Nov 03 09:06:23.605875 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:23.605845    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc4b1919-78b5-46e8-b7ff-2f4a3a048ff3-kube-api-access-xcprx\" (UniqueName: \"kubernetes.io/projected/fc4b1919-78b5-46e8-b7ff-2f4a3a048ff3-kube-api-access-xcprx\") pod \"fc4b1919-78b5-46e8-b7ff-2f4a3a048ff3\" (UID: \"fc4b1919-78b5-46e8-b7ff-2f4a3a048ff3\") "
Nov 03 09:06:23.608022 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:23.608001    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc4b1919-78b5-46e8-b7ff-2f4a3a048ff3-kube-api-access-xcprx" pod "fc4b1919-78b5-46e8-b7ff-2f4a3a048ff3" (UID: "fc4b1919-78b5-46e8-b7ff-2f4a3a048ff3"). InnerVolumeSpecName "kube-api-access-xcprx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:06:23.707563 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:23.707527    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcprx\" (UniqueName: \"kubernetes.io/projected/fc4b1919-78b5-46e8-b7ff-2f4a3a048ff3-kube-api-access-xcprx\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:06:23.870256 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:23.870226    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc4b1919-78b5-46e8-b7ff-2f4a3a048ff3" path="/var/lib/kubelet/pods/fc4b1919-78b5-46e8-b7ff-2f4a3a048ff3/volumes"
Nov 03 09:06:24.210529 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:24.210437    4473 generic.go:356] "Generic (PLEG): container finished" podID="fc4b1919-78b5-46e8-b7ff-2f4a3a048ff3" containerID="df6c4bc4afe5356c62d0c8c29b774879e91acdb8d73da262559c00519b8e5ce7" exitCode=137
Nov 03 09:06:24.210529 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:24.210491    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7378-dceffd"
Nov 03 09:06:24.211009 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:24.210536    4473 scope.go:122] "RemoveContainer" containerID="df6c4bc4afe5356c62d0c8c29b774879e91acdb8d73da262559c00519b8e5ce7"
Nov 03 09:06:24.212382 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:24.212243    4473 generic.go:356] "Generic (PLEG): container finished" podID="fa3aefbe-61d6-476e-9a87-d86fec01ecaf" containerID="86a3a6858ed6adc436573c725e9dcf611d641d21443d2cda69cd7663aa44cde5" exitCode=0
Nov 03 09:06:24.212382 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:24.212276    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b0c653f4-fp44f" event={"ID":"fa3aefbe-61d6-476e-9a87-d86fec01ecaf","Type":"ContainerDied","Data":"86a3a6858ed6adc436573c725e9dcf611d641d21443d2cda69cd7663aa44cde5"}
Nov 03 09:06:24.223646 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:24.223626    4473 scope.go:122] "RemoveContainer" containerID="df6c4bc4afe5356c62d0c8c29b774879e91acdb8d73da262559c00519b8e5ce7"
Nov 03 09:06:24.223936 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:06:24.223914    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df6c4bc4afe5356c62d0c8c29b774879e91acdb8d73da262559c00519b8e5ce7\": container with ID starting with df6c4bc4afe5356c62d0c8c29b774879e91acdb8d73da262559c00519b8e5ce7 not found: ID does not exist" containerID="df6c4bc4afe5356c62d0c8c29b774879e91acdb8d73da262559c00519b8e5ce7"
Nov 03 09:06:24.224037 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:24.223961    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df6c4bc4afe5356c62d0c8c29b774879e91acdb8d73da262559c00519b8e5ce7"} err="failed to get container status \"df6c4bc4afe5356c62d0c8c29b774879e91acdb8d73da262559c00519b8e5ce7\": rpc error: code = NotFound desc = could not find container \"df6c4bc4afe5356c62d0c8c29b774879e91acdb8d73da262559c00519b8e5ce7\": container with ID starting with df6c4bc4afe5356c62d0c8c29b774879e91acdb8d73da262559c00519b8e5ce7 not found: ID does not exist"
Nov 03 09:06:25.378236 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:25.378207    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b0c653f4-fp44f"
Nov 03 09:06:25.427272 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:25.427243    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa3aefbe-61d6-476e-9a87-d86fec01ecaf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa3aefbe-61d6-476e-9a87-d86fec01ecaf-maas-api-xks-ca\") pod \"fa3aefbe-61d6-476e-9a87-d86fec01ecaf\" (UID: \"fa3aefbe-61d6-476e-9a87-d86fec01ecaf\") "
Nov 03 09:06:25.427439 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:25.427394    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa3aefbe-61d6-476e-9a87-d86fec01ecaf-kube-api-access-vjr4x\" (UniqueName: \"kubernetes.io/projected/fa3aefbe-61d6-476e-9a87-d86fec01ecaf-kube-api-access-vjr4x\") pod \"fa3aefbe-61d6-476e-9a87-d86fec01ecaf\" (UID: \"fa3aefbe-61d6-476e-9a87-d86fec01ecaf\") "
Nov 03 09:06:25.427477 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:25.427454    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa3aefbe-61d6-476e-9a87-d86fec01ecaf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa3aefbe-61d6-476e-9a87-d86fec01ecaf-maas-api-openshift-ca\") pod \"fa3aefbe-61d6-476e-9a87-d86fec01ecaf\" (UID: \"fa3aefbe-61d6-476e-9a87-d86fec01ecaf\") "
Nov 03 09:06:25.427853 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:25.427806    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa3aefbe-61d6-476e-9a87-d86fec01ecaf-maas-api-openshift-ca" pod "fa3aefbe-61d6-476e-9a87-d86fec01ecaf" (UID: "fa3aefbe-61d6-476e-9a87-d86fec01ecaf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:06:25.428152 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:25.427926    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa3aefbe-61d6-476e-9a87-d86fec01ecaf-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:06:25.429534 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:25.429511    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa3aefbe-61d6-476e-9a87-d86fec01ecaf-maas-api-xks-ca" pod "fa3aefbe-61d6-476e-9a87-d86fec01ecaf" (UID: "fa3aefbe-61d6-476e-9a87-d86fec01ecaf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:06:25.429675 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:25.429655    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa3aefbe-61d6-476e-9a87-d86fec01ecaf-kube-api-access-vjr4x" pod "fa3aefbe-61d6-476e-9a87-d86fec01ecaf" (UID: "fa3aefbe-61d6-476e-9a87-d86fec01ecaf"). InnerVolumeSpecName "kube-api-access-vjr4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:06:25.529153 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:25.529062    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjr4x\" (UniqueName: \"kubernetes.io/projected/fa3aefbe-61d6-476e-9a87-d86fec01ecaf-kube-api-access-vjr4x\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:06:25.529153 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:25.529103    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa3aefbe-61d6-476e-9a87-d86fec01ecaf-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:06:26.172304 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:26.172212    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d7fd66-7cdbb5785c-zxv4z"]
Nov 03 09:06:26.229457 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:26.229430    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b0c653f4-fp44f"
Nov 03 09:06:26.229722 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:26.229457    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b0c653f4-fp44f" event={"ID":"fa3aefbe-61d6-476e-9a87-d86fec01ecaf","Type":"ContainerDied","Data":"68182f4768ce7fbdbabc18bc5d4283b43211935871a79601215b356d9395930a"}
Nov 03 09:06:26.229868 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:26.229748    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68182f4768ce7fbdbabc18bc5d4283b43211935871a79601215b356d9395930a"
Nov 03 09:06:29.065476 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:29.065438    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-224496da-f7xt7"]
Nov 03 09:06:29.115702 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:29.115672    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-224496da-f7xt7"]
Nov 03 09:06:29.871304 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:29.871274    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e697407f-b6fb-45d7-bc7f-743879e389c5" path="/var/lib/kubelet/pods/e697407f-b6fb-45d7-bc7f-743879e389c5/volumes"
Nov 03 09:06:30.089971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:30.088682    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63d7d166-wrnf4"]
Nov 03 09:06:30.250005 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:30.170048    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63d7d166-wrnf4"]
Nov 03 09:06:31.173112 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:31.173069    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-d7fd66-7cdbb5785c-zxv4z" podUID="f80724ee-b824-4f48-a6af-4c9b8d4c163c" containerName="payload-processing" containerID="cri-o://a17c73144985762de89de3488a53c25f100ae8a35370bfcefd4e506f307e921f" gracePeriod=25
Nov 03 09:06:31.506544 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:31.506520    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d7fd66-7cdbb5785c-zxv4z"
Nov 03 09:06:31.570444 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:31.570404    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f80724ee-b824-4f48-a6af-4c9b8d4c163c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f80724ee-b824-4f48-a6af-4c9b8d4c163c-plugins-config-volume\") pod \"f80724ee-b824-4f48-a6af-4c9b8d4c163c\" (UID: \"f80724ee-b824-4f48-a6af-4c9b8d4c163c\") "
Nov 03 09:06:31.570637 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:31.570569    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f80724ee-b824-4f48-a6af-4c9b8d4c163c-kube-api-access-68pws\" (UniqueName: \"kubernetes.io/projected/f80724ee-b824-4f48-a6af-4c9b8d4c163c-kube-api-access-68pws\") pod \"f80724ee-b824-4f48-a6af-4c9b8d4c163c\" (UID: \"f80724ee-b824-4f48-a6af-4c9b8d4c163c\") "
Nov 03 09:06:31.570859 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:31.570832    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f80724ee-b824-4f48-a6af-4c9b8d4c163c-plugins-config-volume" pod "f80724ee-b824-4f48-a6af-4c9b8d4c163c" (UID: "f80724ee-b824-4f48-a6af-4c9b8d4c163c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:06:31.571152 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:31.571133    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f80724ee-b824-4f48-a6af-4c9b8d4c163c-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:06:31.572849 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:31.572824    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f80724ee-b824-4f48-a6af-4c9b8d4c163c-kube-api-access-68pws" pod "f80724ee-b824-4f48-a6af-4c9b8d4c163c" (UID: "f80724ee-b824-4f48-a6af-4c9b8d4c163c"). InnerVolumeSpecName "kube-api-access-68pws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:06:31.673037 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:31.672995    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68pws\" (UniqueName: \"kubernetes.io/projected/f80724ee-b824-4f48-a6af-4c9b8d4c163c-kube-api-access-68pws\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:06:31.870792 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:31.870759    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee74596e-1d41-4c5e-a156-b3655afab343" path="/var/lib/kubelet/pods/ee74596e-1d41-4c5e-a156-b3655afab343/volumes"
Nov 03 09:06:32.321431 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.321351    4473 generic.go:356] "Generic (PLEG): container finished" podID="f80724ee-b824-4f48-a6af-4c9b8d4c163c" containerID="a17c73144985762de89de3488a53c25f100ae8a35370bfcefd4e506f307e921f" exitCode=0
Nov 03 09:06:32.321431 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.321394    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d7fd66-7cdbb5785c-zxv4z"
Nov 03 09:06:32.321873 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.321439    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d7fd66-7cdbb5785c-zxv4z" event={"ID":"f80724ee-b824-4f48-a6af-4c9b8d4c163c","Type":"ContainerDied","Data":"a17c73144985762de89de3488a53c25f100ae8a35370bfcefd4e506f307e921f"}
Nov 03 09:06:32.321873 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.321486    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d7fd66-7cdbb5785c-zxv4z" event={"ID":"f80724ee-b824-4f48-a6af-4c9b8d4c163c","Type":"ContainerDied","Data":"814edb53599792f484a6a2746900c4a4ee2124aaf2a2b71b64506292bfdae179"}
Nov 03 09:06:32.321873 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.321507    4473 scope.go:122] "RemoveContainer" containerID="a17c73144985762de89de3488a53c25f100ae8a35370bfcefd4e506f307e921f"
Nov 03 09:06:32.331460 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.331437    4473 scope.go:122] "RemoveContainer" containerID="a17c73144985762de89de3488a53c25f100ae8a35370bfcefd4e506f307e921f"
Nov 03 09:06:32.331740 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:06:32.331719    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a17c73144985762de89de3488a53c25f100ae8a35370bfcefd4e506f307e921f\": container with ID starting with a17c73144985762de89de3488a53c25f100ae8a35370bfcefd4e506f307e921f not found: ID does not exist" containerID="a17c73144985762de89de3488a53c25f100ae8a35370bfcefd4e506f307e921f"
Nov 03 09:06:32.331795 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.331747    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a17c73144985762de89de3488a53c25f100ae8a35370bfcefd4e506f307e921f"} err="failed to get container status \"a17c73144985762de89de3488a53c25f100ae8a35370bfcefd4e506f307e921f\": rpc error: code = NotFound desc = could not find container \"a17c73144985762de89de3488a53c25f100ae8a35370bfcefd4e506f307e921f\": container with ID starting with a17c73144985762de89de3488a53c25f100ae8a35370bfcefd4e506f307e921f not found: ID does not exist"
Nov 03 09:06:32.492398 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.492360    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d7fd66-7cdbb5785c-zxv4z"]
Nov 03 09:06:32.553172 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.553138    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d7fd66-7cdbb5785c-zxv4z"]
Nov 03 09:06:32.645869 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.645831    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"]
Nov 03 09:06:32.646451 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.646432    4473 cpu_manager.go:414] "containerMap: removing container" podUID="fa3aefbe-61d6-476e-9a87-d86fec01ecaf" containerName="revoke-keys"
Nov 03 09:06:32.646451 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.646451    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa3aefbe-61d6-476e-9a87-d86fec01ecaf" containerName="revoke-keys"
Nov 03 09:06:32.646687 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.646479    4473 cpu_manager.go:414] "containerMap: removing container" podUID="f80724ee-b824-4f48-a6af-4c9b8d4c163c" containerName="payload-processing"
Nov 03 09:06:32.646687 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.646484    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f80724ee-b824-4f48-a6af-4c9b8d4c163c" containerName="payload-processing"
Nov 03 09:06:32.646687 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.646576    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="f80724ee-b824-4f48-a6af-4c9b8d4c163c" containerName="payload-processing"
Nov 03 09:06:32.646687 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.646592    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa3aefbe-61d6-476e-9a87-d86fec01ecaf" containerName="revoke-keys"
Nov 03 09:06:32.651282 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.651261    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"
Nov 03 09:06:32.691262 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.691219    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5571ff39-4a42-4390-aab3-2f1e60011a4e-istio-podinfo\") pod \"e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4\" (UID: \"5571ff39-4a42-4390-aab3-2f1e60011a4e\") " pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"
Nov 03 09:06:32.691572 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.691546    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5571ff39-4a42-4390-aab3-2f1e60011a4e-istiod-ca-cert\") pod \"e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4\" (UID: \"5571ff39-4a42-4390-aab3-2f1e60011a4e\") " pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"
Nov 03 09:06:32.691758 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.691737    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5571ff39-4a42-4390-aab3-2f1e60011a4e-istio-token\") pod \"e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4\" (UID: \"5571ff39-4a42-4390-aab3-2f1e60011a4e\") " pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"
Nov 03 09:06:32.691973 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.691934    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5571ff39-4a42-4390-aab3-2f1e60011a4e-workload-certs\") pod \"e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4\" (UID: \"5571ff39-4a42-4390-aab3-2f1e60011a4e\") " pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"
Nov 03 09:06:32.692116 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.692099    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2896\" (UniqueName: \"kubernetes.io/projected/5571ff39-4a42-4390-aab3-2f1e60011a4e-kube-api-access-b2896\") pod \"e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4\" (UID: \"5571ff39-4a42-4390-aab3-2f1e60011a4e\") " pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"
Nov 03 09:06:32.692356 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.692337    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5571ff39-4a42-4390-aab3-2f1e60011a4e-istio-envoy\") pod \"e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4\" (UID: \"5571ff39-4a42-4390-aab3-2f1e60011a4e\") " pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"
Nov 03 09:06:32.692474 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.692458    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5571ff39-4a42-4390-aab3-2f1e60011a4e-istio-data\") pod \"e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4\" (UID: \"5571ff39-4a42-4390-aab3-2f1e60011a4e\") " pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"
Nov 03 09:06:32.692708 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.692691    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5571ff39-4a42-4390-aab3-2f1e60011a4e-workload-socket\") pod \"e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4\" (UID: \"5571ff39-4a42-4390-aab3-2f1e60011a4e\") " pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"
Nov 03 09:06:32.695056 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.695031    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5571ff39-4a42-4390-aab3-2f1e60011a4e-credential-socket\") pod \"e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4\" (UID: \"5571ff39-4a42-4390-aab3-2f1e60011a4e\") " pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"
Nov 03 09:06:32.721972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.711502    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-04934e-openshift-default-dockercfg-w2796\""
Nov 03 09:06:32.752374 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.750371    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"]
Nov 03 09:06:32.833315 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.833277    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5571ff39-4a42-4390-aab3-2f1e60011a4e-istio-envoy\") pod \"e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4\" (UID: \"5571ff39-4a42-4390-aab3-2f1e60011a4e\") " pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"
Nov 03 09:06:32.833553 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.833529    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5571ff39-4a42-4390-aab3-2f1e60011a4e-istio-data\") pod \"e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4\" (UID: \"5571ff39-4a42-4390-aab3-2f1e60011a4e\") " pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"
Nov 03 09:06:32.833781 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.833761    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5571ff39-4a42-4390-aab3-2f1e60011a4e-workload-socket\") pod \"e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4\" (UID: \"5571ff39-4a42-4390-aab3-2f1e60011a4e\") " pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"
Nov 03 09:06:32.833922 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.833907    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5571ff39-4a42-4390-aab3-2f1e60011a4e-credential-socket\") pod \"e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4\" (UID: \"5571ff39-4a42-4390-aab3-2f1e60011a4e\") " pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"
Nov 03 09:06:32.847738 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.847682    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5571ff39-4a42-4390-aab3-2f1e60011a4e-istio-podinfo\") pod \"e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4\" (UID: \"5571ff39-4a42-4390-aab3-2f1e60011a4e\") " pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"
Nov 03 09:06:32.848313 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.848265    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5571ff39-4a42-4390-aab3-2f1e60011a4e-istiod-ca-cert\") pod \"e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4\" (UID: \"5571ff39-4a42-4390-aab3-2f1e60011a4e\") " pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"
Nov 03 09:06:32.848589 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.848568    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5571ff39-4a42-4390-aab3-2f1e60011a4e-istio-token\") pod \"e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4\" (UID: \"5571ff39-4a42-4390-aab3-2f1e60011a4e\") " pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"
Nov 03 09:06:32.902549 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.900162    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5571ff39-4a42-4390-aab3-2f1e60011a4e-workload-certs\") pod \"e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4\" (UID: \"5571ff39-4a42-4390-aab3-2f1e60011a4e\") " pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"
Nov 03 09:06:32.902549 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.900221    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2896\" (UniqueName: \"kubernetes.io/projected/5571ff39-4a42-4390-aab3-2f1e60011a4e-kube-api-access-b2896\") pod \"e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4\" (UID: \"5571ff39-4a42-4390-aab3-2f1e60011a4e\") " pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"
Nov 03 09:06:32.966797 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.928348    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5571ff39-4a42-4390-aab3-2f1e60011a4e-istio-data\") pod \"e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4\" (UID: \"5571ff39-4a42-4390-aab3-2f1e60011a4e\") " pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"
Nov 03 09:06:32.966797 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.964177    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5571ff39-4a42-4390-aab3-2f1e60011a4e-istio-podinfo\") pod \"e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4\" (UID: \"5571ff39-4a42-4390-aab3-2f1e60011a4e\") " pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"
Nov 03 09:06:32.966797 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.966481    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5571ff39-4a42-4390-aab3-2f1e60011a4e-istiod-ca-cert\") pod \"e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4\" (UID: \"5571ff39-4a42-4390-aab3-2f1e60011a4e\") " pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"
Nov 03 09:06:32.966797 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.966722    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5571ff39-4a42-4390-aab3-2f1e60011a4e-credential-socket\") pod \"e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4\" (UID: \"5571ff39-4a42-4390-aab3-2f1e60011a4e\") " pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"
Nov 03 09:06:32.967373 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.967309    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5571ff39-4a42-4390-aab3-2f1e60011a4e-workload-certs\") pod \"e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4\" (UID: \"5571ff39-4a42-4390-aab3-2f1e60011a4e\") " pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"
Nov 03 09:06:32.967625 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:32.967605    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5571ff39-4a42-4390-aab3-2f1e60011a4e-workload-socket\") pod \"e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4\" (UID: \"5571ff39-4a42-4390-aab3-2f1e60011a4e\") " pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"
Nov 03 09:06:33.001095 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:33.001004    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2896\" (UniqueName: \"kubernetes.io/projected/5571ff39-4a42-4390-aab3-2f1e60011a4e-kube-api-access-b2896\") pod \"e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4\" (UID: \"5571ff39-4a42-4390-aab3-2f1e60011a4e\") " pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"
Nov 03 09:06:33.049974 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:33.046517    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5571ff39-4a42-4390-aab3-2f1e60011a4e-istio-envoy\") pod \"e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4\" (UID: \"5571ff39-4a42-4390-aab3-2f1e60011a4e\") " pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"
Nov 03 09:06:33.050178 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:33.050104    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5571ff39-4a42-4390-aab3-2f1e60011a4e-istio-token\") pod \"e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4\" (UID: \"5571ff39-4a42-4390-aab3-2f1e60011a4e\") " pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"
Nov 03 09:06:33.269662 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:33.269576    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"
Nov 03 09:06:33.763969 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:33.761354    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"]
Nov 03 09:06:33.813969 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:06:33.812675    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5571ff39_4a42_4390_aab3_2f1e60011a4e.slice/crio-3e84375968fd17649647cf70da9acd7424d0467f6fc2682a6abb22639ae9dba1 WatchSource:0}: Error finding container 3e84375968fd17649647cf70da9acd7424d0467f6fc2682a6abb22639ae9dba1: Status 404 returned error can't find the container with id 3e84375968fd17649647cf70da9acd7424d0467f6fc2682a6abb22639ae9dba1
Nov 03 09:06:33.829619 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:33.829576    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:06:33.829817 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:33.829670    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:06:33.829817 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:33.829717    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:06:33.893724 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:33.888912    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f80724ee-b824-4f48-a6af-4c9b8d4c163c" path="/var/lib/kubelet/pods/f80724ee-b824-4f48-a6af-4c9b8d4c163c/volumes"
Nov 03 09:06:34.364579 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:34.364477    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4" event={"ID":"5571ff39-4a42-4390-aab3-2f1e60011a4e","Type":"ContainerStarted","Data":"f7812c462d79f52797cef5c89402e0d737854f743ce110893ab312c840d0e0b1"}
Nov 03 09:06:34.364579 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:34.364549    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4" event={"ID":"5571ff39-4a42-4390-aab3-2f1e60011a4e","Type":"ContainerStarted","Data":"3e84375968fd17649647cf70da9acd7424d0467f6fc2682a6abb22639ae9dba1"}
Nov 03 09:06:34.492404 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:34.492350    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4" podStartSLOduration=2.49233493 podStartE2EDuration="2.49233493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:06:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:06:34.489318053 +0000 UTC m=+1618.758381501" watchObservedRunningTime="2026-11-03 09:06:34.49233493 +0000 UTC m=+1618.761398365"
Nov 03 09:06:35.054700 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:35.054662    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a40cf0f3-6c6z8"]
Nov 03 09:06:35.060589 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:35.060564    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a40cf0f3-6c6z8"]
Nov 03 09:06:35.270219 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:35.270187    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"
Nov 03 09:06:35.275813 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:35.275784    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"
Nov 03 09:06:35.371981 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:35.371939    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"
Nov 03 09:06:35.372844 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:35.372822    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"
Nov 03 09:06:35.870436 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:35.870409    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92df7d0e-c794-4e54-b75f-0d238d209d16" path="/var/lib/kubelet/pods/92df7d0e-c794-4e54-b75f-0d238d209d16/volumes"
Nov 03 09:06:36.426029 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:36.426003    4473 scope.go:122] "RemoveContainer" containerID="0e8ade94451328b89152a3d2226b90a71bf2ac2feeee333da233c1c801015404"
Nov 03 09:06:37.689375 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:37.689331    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-840559-798b95d567-46skj"]
Nov 03 09:06:37.712889 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:37.700184    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-840559-798b95d567-46skj"
Nov 03 09:06:37.761867 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:37.761821    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-840559-798b95d567-46skj"]
Nov 03 09:06:37.813909 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:37.813876    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-840559-dockercfg-kg87z\""
Nov 03 09:06:37.814628 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:37.814591    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-840559\""
Nov 03 09:06:37.816138 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:37.816117    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ph74\" (UniqueName: \"kubernetes.io/projected/048badaf-8a93-4e51-8308-2a75a72487f3-kube-api-access-8ph74\") pod \"payload-pre-processing-e2e-api-a-840559-798b95d567-46skj\" (UID: \"048badaf-8a93-4e51-8308-2a75a72487f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-840559-798b95d567-46skj"
Nov 03 09:06:37.816450 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:37.816429    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048badaf-8a93-4e51-8308-2a75a72487f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-840559-798b95d567-46skj\" (UID: \"048badaf-8a93-4e51-8308-2a75a72487f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-840559-798b95d567-46skj"
Nov 03 09:06:37.936974 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:37.923478    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048badaf-8a93-4e51-8308-2a75a72487f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-840559-798b95d567-46skj\" (UID: \"048badaf-8a93-4e51-8308-2a75a72487f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-840559-798b95d567-46skj"
Nov 03 09:06:37.936974 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:37.923804    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ph74\" (UniqueName: \"kubernetes.io/projected/048badaf-8a93-4e51-8308-2a75a72487f3-kube-api-access-8ph74\") pod \"payload-pre-processing-e2e-api-a-840559-798b95d567-46skj\" (UID: \"048badaf-8a93-4e51-8308-2a75a72487f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-840559-798b95d567-46skj"
Nov 03 09:06:37.936974 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:37.931913    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048badaf-8a93-4e51-8308-2a75a72487f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-840559-798b95d567-46skj\" (UID: \"048badaf-8a93-4e51-8308-2a75a72487f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-840559-798b95d567-46skj"
Nov 03 09:06:37.999683 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:37.999605    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ph74\" (UniqueName: \"kubernetes.io/projected/048badaf-8a93-4e51-8308-2a75a72487f3-kube-api-access-8ph74\") pod \"payload-pre-processing-e2e-api-a-840559-798b95d567-46skj\" (UID: \"048badaf-8a93-4e51-8308-2a75a72487f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-840559-798b95d567-46skj"
Nov 03 09:06:38.041472 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:38.041445    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-840559-798b95d567-46skj"
Nov 03 09:06:38.419193 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:38.419163    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-840559-798b95d567-46skj"]
Nov 03 09:06:38.419553 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:06:38.419531    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod048badaf_8a93_4e51_8308_2a75a72487f3.slice/crio-61d9479b83960710e7f72c6249669676fc243dfca0e4a8e5af175698dbb72e4b WatchSource:0}: Error finding container 61d9479b83960710e7f72c6249669676fc243dfca0e4a8e5af175698dbb72e4b: Status 404 returned error can't find the container with id 61d9479b83960710e7f72c6249669676fc243dfca0e4a8e5af175698dbb72e4b
Nov 03 09:06:38.490499 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:38.490477    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-840559-798b95d567-46skj" event={"ID":"048badaf-8a93-4e51-8308-2a75a72487f3","Type":"ContainerStarted","Data":"61d9479b83960710e7f72c6249669676fc243dfca0e4a8e5af175698dbb72e4b"}
Nov 03 09:06:39.498930 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:39.498895    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-840559-798b95d567-46skj" event={"ID":"048badaf-8a93-4e51-8308-2a75a72487f3","Type":"ContainerStarted","Data":"9ed25b860de7619100afdd4583fd9179c90749b3d7c802eff036b45c4a13e2ef"}
Nov 03 09:06:39.499339 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:39.499060    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-840559-798b95d567-46skj"
Nov 03 09:06:39.618083 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:39.618029    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-840559-798b95d567-46skj" podStartSLOduration=2.618011134 podStartE2EDuration="2.618011134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:06:39.614356886 +0000 UTC m=+1623.883420344" watchObservedRunningTime="2026-11-03 09:06:39.618011134 +0000 UTC m=+1623.887074569"
Nov 03 09:06:42.752903 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:42.752863    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-7cd97b9c75-dhs4m"]
Nov 03 09:06:42.762671 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:42.762646    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-7cd97b9c75-dhs4m"
Nov 03 09:06:42.798406 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:42.798370    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-7cd97b9c75-dhs4m"]
Nov 03 09:06:42.837448 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:42.835166    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02d38dda-9627-4c78-8b4e-4257f3771afc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-04934e-7cd97b9c75-dhs4m\" (UID: \"02d38dda-9627-4c78-8b4e-4257f3771afc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-7cd97b9c75-dhs4m"
Nov 03 09:06:42.837448 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:42.835269    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrgsv\" (UniqueName: \"kubernetes.io/projected/02d38dda-9627-4c78-8b4e-4257f3771afc-kube-api-access-wrgsv\") pod \"payload-pre-processing-e2e-ipp-a-04934e-7cd97b9c75-dhs4m\" (UID: \"02d38dda-9627-4c78-8b4e-4257f3771afc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-7cd97b9c75-dhs4m"
Nov 03 09:06:42.852970 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:42.848127    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-04934e\""
Nov 03 09:06:42.852970 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:42.848380    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-04934e-dockercfg-xvjm5\""
Nov 03 09:06:42.991917 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:42.987844    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02d38dda-9627-4c78-8b4e-4257f3771afc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-04934e-7cd97b9c75-dhs4m\" (UID: \"02d38dda-9627-4c78-8b4e-4257f3771afc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-7cd97b9c75-dhs4m"
Nov 03 09:06:42.992199 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:42.974937    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02d38dda-9627-4c78-8b4e-4257f3771afc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-04934e-7cd97b9c75-dhs4m\" (UID: \"02d38dda-9627-4c78-8b4e-4257f3771afc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-7cd97b9c75-dhs4m"
Nov 03 09:06:42.992419 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:42.992396    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrgsv\" (UniqueName: \"kubernetes.io/projected/02d38dda-9627-4c78-8b4e-4257f3771afc-kube-api-access-wrgsv\") pod \"payload-pre-processing-e2e-ipp-a-04934e-7cd97b9c75-dhs4m\" (UID: \"02d38dda-9627-4c78-8b4e-4257f3771afc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-7cd97b9c75-dhs4m"
Nov 03 09:06:43.053127 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:43.053055    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrgsv\" (UniqueName: \"kubernetes.io/projected/02d38dda-9627-4c78-8b4e-4257f3771afc-kube-api-access-wrgsv\") pod \"payload-pre-processing-e2e-ipp-a-04934e-7cd97b9c75-dhs4m\" (UID: \"02d38dda-9627-4c78-8b4e-4257f3771afc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-7cd97b9c75-dhs4m"
Nov 03 09:06:43.095912 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:43.095883    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-7cd97b9c75-dhs4m"
Nov 03 09:06:43.474151 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:43.474120    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-7cd97b9c75-dhs4m"]
Nov 03 09:06:43.474290 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:06:43.474268    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02d38dda_9627_4c78_8b4e_4257f3771afc.slice/crio-4e5115f9d53cb3537903cb0d15291dbdbc7a024867674972fe0e134b3fbb2cd2 WatchSource:0}: Error finding container 4e5115f9d53cb3537903cb0d15291dbdbc7a024867674972fe0e134b3fbb2cd2: Status 404 returned error can't find the container with id 4e5115f9d53cb3537903cb0d15291dbdbc7a024867674972fe0e134b3fbb2cd2
Nov 03 09:06:43.581316 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:43.581279    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-7cd97b9c75-dhs4m" event={"ID":"02d38dda-9627-4c78-8b4e-4257f3771afc","Type":"ContainerStarted","Data":"ad4860276f954c6301d5f7095631d66dfe99c550cbbaf2ed15c37ad4723552fc"}
Nov 03 09:06:43.581486 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:43.581334    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-7cd97b9c75-dhs4m" event={"ID":"02d38dda-9627-4c78-8b4e-4257f3771afc","Type":"ContainerStarted","Data":"4e5115f9d53cb3537903cb0d15291dbdbc7a024867674972fe0e134b3fbb2cd2"}
Nov 03 09:06:43.581546 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:43.581497    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-7cd97b9c75-dhs4m"
Nov 03 09:06:43.700007 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:43.699956    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-7cd97b9c75-dhs4m" podStartSLOduration=1.6999277959999999 podStartE2EDuration="1.699927796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:06:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:06:43.697205107 +0000 UTC m=+1627.966268564" watchObservedRunningTime="2026-11-03 09:06:43.699927796 +0000 UTC m=+1627.968991229"
Nov 03 09:06:46.937645 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:06:46.937610    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:06:48.054137 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:48.054103    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c1d945a-lccxp"]
Nov 03 09:06:48.104542 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:48.104512    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c1d945a-lccxp"]
Nov 03 09:06:49.879466 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:49.879082    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0980c8ec-9657-45a6-bd54-d78be27d7b32" path="/var/lib/kubelet/pods/0980c8ec-9657-45a6-bd54-d78be27d7b32/volumes"
Nov 03 09:06:50.521380 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:50.521351    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-840559-798b95d567-46skj"
Nov 03 09:06:54.591467 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:54.591431    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-7cd97b9c75-dhs4m"
Nov 03 09:06:57.055018 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:57.054982    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0803d9e-d255b"]
Nov 03 09:06:57.106304 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:57.106277    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0803d9e-d255b"]
Nov 03 09:06:57.870727 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:57.870693    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66c905c8-ee01-442a-ab3d-27167deb93ce" path="/var/lib/kubelet/pods/66c905c8-ee01-442a-ab3d-27167deb93ce/volumes"
Nov 03 09:06:59.877152 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:06:59.877124    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:07:00.092842 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:00.092808    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b8fd34-6d55957755-c955l"]
Nov 03 09:07:05.093147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:05.093108    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-b8fd34-6d55957755-c955l" podUID="83047914-9b8f-4ea3-9b3e-caea9f8dcaa1" containerName="payload-processing" containerID="cri-o://a2e9dd25cf9b00a65469ed8c47f899360fdee3bd7562e434c14f9b37c59dee60" gracePeriod=25
Nov 03 09:07:05.439380 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:05.439343    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b8fd34-6d55957755-c955l"
Nov 03 09:07:05.562501 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:05.562467    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83047914-9b8f-4ea3-9b3e-caea9f8dcaa1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83047914-9b8f-4ea3-9b3e-caea9f8dcaa1-plugins-config-volume\") pod \"83047914-9b8f-4ea3-9b3e-caea9f8dcaa1\" (UID: \"83047914-9b8f-4ea3-9b3e-caea9f8dcaa1\") "
Nov 03 09:07:05.562686 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:05.562538    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83047914-9b8f-4ea3-9b3e-caea9f8dcaa1-kube-api-access-p8xn7\" (UniqueName: \"kubernetes.io/projected/83047914-9b8f-4ea3-9b3e-caea9f8dcaa1-kube-api-access-p8xn7\") pod \"83047914-9b8f-4ea3-9b3e-caea9f8dcaa1\" (UID: \"83047914-9b8f-4ea3-9b3e-caea9f8dcaa1\") "
Nov 03 09:07:05.562913 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:05.562877    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83047914-9b8f-4ea3-9b3e-caea9f8dcaa1-plugins-config-volume" pod "83047914-9b8f-4ea3-9b3e-caea9f8dcaa1" (UID: "83047914-9b8f-4ea3-9b3e-caea9f8dcaa1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:07:05.563146 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:05.563127    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83047914-9b8f-4ea3-9b3e-caea9f8dcaa1-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:07:05.565580 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:05.565555    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83047914-9b8f-4ea3-9b3e-caea9f8dcaa1-kube-api-access-p8xn7" pod "83047914-9b8f-4ea3-9b3e-caea9f8dcaa1" (UID: "83047914-9b8f-4ea3-9b3e-caea9f8dcaa1"). InnerVolumeSpecName "kube-api-access-p8xn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:07:05.664749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:05.664665    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8xn7\" (UniqueName: \"kubernetes.io/projected/83047914-9b8f-4ea3-9b3e-caea9f8dcaa1-kube-api-access-p8xn7\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:07:05.812817 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:05.812777    4473 generic.go:356] "Generic (PLEG): container finished" podID="83047914-9b8f-4ea3-9b3e-caea9f8dcaa1" containerID="a2e9dd25cf9b00a65469ed8c47f899360fdee3bd7562e434c14f9b37c59dee60" exitCode=0
Nov 03 09:07:05.813023 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:05.812823    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b8fd34-6d55957755-c955l"
Nov 03 09:07:05.813023 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:05.812829    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b8fd34-6d55957755-c955l" event={"ID":"83047914-9b8f-4ea3-9b3e-caea9f8dcaa1","Type":"ContainerDied","Data":"a2e9dd25cf9b00a65469ed8c47f899360fdee3bd7562e434c14f9b37c59dee60"}
Nov 03 09:07:05.813023 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:05.812880    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b8fd34-6d55957755-c955l" event={"ID":"83047914-9b8f-4ea3-9b3e-caea9f8dcaa1","Type":"ContainerDied","Data":"1f849eb4970d0f93ba5559dc08254b570e3aa6d6f357900ea58ed9149f69f2e0"}
Nov 03 09:07:05.813023 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:05.812939    4473 scope.go:122] "RemoveContainer" containerID="a2e9dd25cf9b00a65469ed8c47f899360fdee3bd7562e434c14f9b37c59dee60"
Nov 03 09:07:05.823254 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:05.823237    4473 scope.go:122] "RemoveContainer" containerID="a2e9dd25cf9b00a65469ed8c47f899360fdee3bd7562e434c14f9b37c59dee60"
Nov 03 09:07:05.823533 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:07:05.823515    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2e9dd25cf9b00a65469ed8c47f899360fdee3bd7562e434c14f9b37c59dee60\": container with ID starting with a2e9dd25cf9b00a65469ed8c47f899360fdee3bd7562e434c14f9b37c59dee60 not found: ID does not exist" containerID="a2e9dd25cf9b00a65469ed8c47f899360fdee3bd7562e434c14f9b37c59dee60"
Nov 03 09:07:05.823597 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:05.823546    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2e9dd25cf9b00a65469ed8c47f899360fdee3bd7562e434c14f9b37c59dee60"} err="failed to get container status \"a2e9dd25cf9b00a65469ed8c47f899360fdee3bd7562e434c14f9b37c59dee60\": rpc error: code = NotFound desc = could not find container \"a2e9dd25cf9b00a65469ed8c47f899360fdee3bd7562e434c14f9b37c59dee60\": container with ID starting with a2e9dd25cf9b00a65469ed8c47f899360fdee3bd7562e434c14f9b37c59dee60 not found: ID does not exist"
Nov 03 09:07:05.989226 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:05.989144    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b8fd34-6d55957755-c955l"]
Nov 03 09:07:05.994255 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:05.994224    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b8fd34-6d55957755-c955l"]
Nov 03 09:07:07.881962 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:07.880005    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83047914-9b8f-4ea3-9b3e-caea9f8dcaa1" path="/var/lib/kubelet/pods/83047914-9b8f-4ea3-9b3e-caea9f8dcaa1/volumes"
Nov 03 09:07:16.054172 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:16.054140    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d1ebae5-rvxks"]
Nov 03 09:07:16.104228 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:16.104200    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d1ebae5-rvxks"]
Nov 03 09:07:16.937326 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:07:16.937239    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:07:17.870410 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:17.870377    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd482794-363d-4374-a3a5-e0848799005e" path="/var/lib/kubelet/pods/dd482794-363d-4374-a3a5-e0848799005e/volumes"
Nov 03 09:07:17.957964 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:17.957922    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da7f4c25-cwtx8"]
Nov 03 09:07:17.958782 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:17.958766    4473 cpu_manager.go:414] "containerMap: removing container" podUID="83047914-9b8f-4ea3-9b3e-caea9f8dcaa1" containerName="payload-processing"
Nov 03 09:07:17.958832 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:17.958786    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83047914-9b8f-4ea3-9b3e-caea9f8dcaa1" containerName="payload-processing"
Nov 03 09:07:17.958925 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:17.958914    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="83047914-9b8f-4ea3-9b3e-caea9f8dcaa1" containerName="payload-processing"
Nov 03 09:07:17.964300 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:17.964277    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7f4c25-cwtx8"
Nov 03 09:07:18.021330 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:18.021301    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da7f4c25-cwtx8"]
Nov 03 09:07:18.024149 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:18.024124    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b64db0-8da5-4bb5-a1dc-d4962912ad9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da7f4c25-cwtx8\" (UID: \"d9b64db0-8da5-4bb5-a1dc-d4962912ad9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7f4c25-cwtx8"
Nov 03 09:07:18.024272 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:18.024172    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxnck\" (UniqueName: \"kubernetes.io/projected/d9b64db0-8da5-4bb5-a1dc-d4962912ad9f-kube-api-access-bxnck\") pod \"maas-api-revoke-sub-da7f4c25-cwtx8\" (UID: \"d9b64db0-8da5-4bb5-a1dc-d4962912ad9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7f4c25-cwtx8"
Nov 03 09:07:18.024337 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:18.024298    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b64db0-8da5-4bb5-a1dc-d4962912ad9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da7f4c25-cwtx8\" (UID: \"d9b64db0-8da5-4bb5-a1dc-d4962912ad9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7f4c25-cwtx8"
Nov 03 09:07:18.066996 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:18.066965    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:07:18.066996 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:18.066965    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:07:18.067171 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:18.067004    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:07:18.068484 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:18.068468    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:07:18.125050 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:18.124988    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b64db0-8da5-4bb5-a1dc-d4962912ad9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da7f4c25-cwtx8\" (UID: \"d9b64db0-8da5-4bb5-a1dc-d4962912ad9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7f4c25-cwtx8"
Nov 03 09:07:18.125230 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:18.125211    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b64db0-8da5-4bb5-a1dc-d4962912ad9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da7f4c25-cwtx8\" (UID: \"d9b64db0-8da5-4bb5-a1dc-d4962912ad9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7f4c25-cwtx8"
Nov 03 09:07:18.125272 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:18.125261    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxnck\" (UniqueName: \"kubernetes.io/projected/d9b64db0-8da5-4bb5-a1dc-d4962912ad9f-kube-api-access-bxnck\") pod \"maas-api-revoke-sub-da7f4c25-cwtx8\" (UID: \"d9b64db0-8da5-4bb5-a1dc-d4962912ad9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7f4c25-cwtx8"
Nov 03 09:07:18.125888 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:18.125872    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b64db0-8da5-4bb5-a1dc-d4962912ad9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da7f4c25-cwtx8\" (UID: \"d9b64db0-8da5-4bb5-a1dc-d4962912ad9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7f4c25-cwtx8"
Nov 03 09:07:18.127637 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:18.127616    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b64db0-8da5-4bb5-a1dc-d4962912ad9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da7f4c25-cwtx8\" (UID: \"d9b64db0-8da5-4bb5-a1dc-d4962912ad9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7f4c25-cwtx8"
Nov 03 09:07:18.185392 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:18.185360    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxnck\" (UniqueName: \"kubernetes.io/projected/d9b64db0-8da5-4bb5-a1dc-d4962912ad9f-kube-api-access-bxnck\") pod \"maas-api-revoke-sub-da7f4c25-cwtx8\" (UID: \"d9b64db0-8da5-4bb5-a1dc-d4962912ad9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7f4c25-cwtx8"
Nov 03 09:07:18.278189 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:18.278155    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7f4c25-cwtx8"
Nov 03 09:07:18.660439 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:18.660409    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da7f4c25-cwtx8"]
Nov 03 09:07:18.661992 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:07:18.661920    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9b64db0_8da5_4bb5_a1dc_d4962912ad9f.slice/crio-6ae627c4cfdd1a16264d691802f7824eada33296cbf3a5059ed638b1a5be6bc0 WatchSource:0}: Error finding container 6ae627c4cfdd1a16264d691802f7824eada33296cbf3a5059ed638b1a5be6bc0: Status 404 returned error can't find the container with id 6ae627c4cfdd1a16264d691802f7824eada33296cbf3a5059ed638b1a5be6bc0
Nov 03 09:07:18.918183 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:18.918106    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7f4c25-cwtx8" event={"ID":"d9b64db0-8da5-4bb5-a1dc-d4962912ad9f","Type":"ContainerStarted","Data":"6ae627c4cfdd1a16264d691802f7824eada33296cbf3a5059ed638b1a5be6bc0"}
Nov 03 09:07:19.927907 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:19.927871    4473 generic.go:356] "Generic (PLEG): container finished" podID="d9b64db0-8da5-4bb5-a1dc-d4962912ad9f" containerID="aaebe709a5f6fc5e52fcfcdd99680f2ceb9b6c59c8b208d18e1764cfcc1ca351" exitCode=0
Nov 03 09:07:19.928331 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:19.927964    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7f4c25-cwtx8" event={"ID":"d9b64db0-8da5-4bb5-a1dc-d4962912ad9f","Type":"ContainerDied","Data":"aaebe709a5f6fc5e52fcfcdd99680f2ceb9b6c59c8b208d18e1764cfcc1ca351"}
Nov 03 09:07:21.097804 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:21.097778    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7f4c25-cwtx8"
Nov 03 09:07:21.167258 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:21.167236    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9b64db0-8da5-4bb5-a1dc-d4962912ad9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b64db0-8da5-4bb5-a1dc-d4962912ad9f-maas-api-xks-ca\") pod \"d9b64db0-8da5-4bb5-a1dc-d4962912ad9f\" (UID: \"d9b64db0-8da5-4bb5-a1dc-d4962912ad9f\") "
Nov 03 09:07:21.167415 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:21.167356    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9b64db0-8da5-4bb5-a1dc-d4962912ad9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b64db0-8da5-4bb5-a1dc-d4962912ad9f-maas-api-openshift-ca\") pod \"d9b64db0-8da5-4bb5-a1dc-d4962912ad9f\" (UID: \"d9b64db0-8da5-4bb5-a1dc-d4962912ad9f\") "
Nov 03 09:07:21.167500 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:21.167420    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9b64db0-8da5-4bb5-a1dc-d4962912ad9f-kube-api-access-bxnck\" (UniqueName: \"kubernetes.io/projected/d9b64db0-8da5-4bb5-a1dc-d4962912ad9f-kube-api-access-bxnck\") pod \"d9b64db0-8da5-4bb5-a1dc-d4962912ad9f\" (UID: \"d9b64db0-8da5-4bb5-a1dc-d4962912ad9f\") "
Nov 03 09:07:21.167688 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:21.167665    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9b64db0-8da5-4bb5-a1dc-d4962912ad9f-maas-api-openshift-ca" pod "d9b64db0-8da5-4bb5-a1dc-d4962912ad9f" (UID: "d9b64db0-8da5-4bb5-a1dc-d4962912ad9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:07:21.167807 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:21.167793    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9b64db0-8da5-4bb5-a1dc-d4962912ad9f-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:07:21.169550 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:21.169529    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9b64db0-8da5-4bb5-a1dc-d4962912ad9f-kube-api-access-bxnck" pod "d9b64db0-8da5-4bb5-a1dc-d4962912ad9f" (UID: "d9b64db0-8da5-4bb5-a1dc-d4962912ad9f"). InnerVolumeSpecName "kube-api-access-bxnck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:07:21.169698 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:21.169682    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9b64db0-8da5-4bb5-a1dc-d4962912ad9f-maas-api-xks-ca" pod "d9b64db0-8da5-4bb5-a1dc-d4962912ad9f" (UID: "d9b64db0-8da5-4bb5-a1dc-d4962912ad9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:07:21.269503 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:21.269469    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxnck\" (UniqueName: \"kubernetes.io/projected/d9b64db0-8da5-4bb5-a1dc-d4962912ad9f-kube-api-access-bxnck\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:07:21.269503 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:21.269497    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9b64db0-8da5-4bb5-a1dc-d4962912ad9f-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:07:21.946284 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:21.946248    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7f4c25-cwtx8" event={"ID":"d9b64db0-8da5-4bb5-a1dc-d4962912ad9f","Type":"ContainerDied","Data":"6ae627c4cfdd1a16264d691802f7824eada33296cbf3a5059ed638b1a5be6bc0"}
Nov 03 09:07:21.946455 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:21.946303    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ae627c4cfdd1a16264d691802f7824eada33296cbf3a5059ed638b1a5be6bc0"
Nov 03 09:07:21.946455 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:21.946263    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da7f4c25-cwtx8"
Nov 03 09:07:25.052195 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:25.052151    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-140e33df-7dvl7"]
Nov 03 09:07:25.102502 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:25.102470    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-140e33df-7dvl7"]
Nov 03 09:07:25.871290 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:25.871257    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21322b5d-c21e-4ecf-8990-c964de95e28e" path="/var/lib/kubelet/pods/21322b5d-c21e-4ecf-8990-c964de95e28e/volumes"
Nov 03 09:07:27.053694 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:27.053660    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24cd268f-8tzfp"]
Nov 03 09:07:27.104406 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:27.104381    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24cd268f-8tzfp"]
Nov 03 09:07:27.870856 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:27.870825    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b69a3f01-23ee-4433-85ad-c3e5300e58ff" path="/var/lib/kubelet/pods/b69a3f01-23ee-4433-85ad-c3e5300e58ff/volumes"
Nov 03 09:07:30.138662 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:30.138630    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:07:36.463256 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:36.463223    4473 scope.go:122] "RemoveContainer" containerID="be7e763f0e6042b8fc634aca61f7dcf8d7ddd4bcdf1be1539b6bc9c22ed329c2"
Nov 03 09:07:36.473095 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:36.473069    4473 scope.go:122] "RemoveContainer" containerID="f5925db1ccbd921e3ae4021c47301bf3a1de9a7fabb923a6d63449574c43e9ca"
Nov 03 09:07:36.481880 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:36.481860    4473 scope.go:122] "RemoveContainer" containerID="fae32803d23e9a04a827800091182dd81863f1692e47e3cf6f61591af563f797"
Nov 03 09:07:36.490817 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:36.490799    4473 scope.go:122] "RemoveContainer" containerID="d366fa85856c8ac7d0cfb90d17669ccd9bb9acc83073daffb9c4ea8ead046c37"
Nov 03 09:07:36.499792 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:36.499775    4473 scope.go:122] "RemoveContainer" containerID="d1c7755d0d14f058f75b73470d5d238917b4e1dc76ecf55bd82795f54a4a141a"
Nov 03 09:07:36.508356 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:36.508338    4473 scope.go:122] "RemoveContainer" containerID="350f5f16eb6acbf239cd68905016614fb82cd7f980d4793bb56223a54b77ec40"
Nov 03 09:07:36.517706 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:36.517685    4473 scope.go:122] "RemoveContainer" containerID="058a1b8533506f0403f410a5009bd9ed90f9649800baf63c0ab9fec2dc1055c8"
Nov 03 09:07:36.528639 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:36.528623    4473 scope.go:122] "RemoveContainer" containerID="efe1df3cd7675b92123e2e237a178bd897d197ab249ad4c597e216997da5db55"
Nov 03 09:07:40.127864 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:40.127835    4473 generic.go:356] "Generic (PLEG): container finished" podID="4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5" containerID="a3d29a7c4944cc360ce95640deabd3a70173b062a18f843efad97f5e894c3b01" exitCode=0
Nov 03 09:07:40.128272 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:40.127909    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w" event={"ID":"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5","Type":"ContainerDied","Data":"a3d29a7c4944cc360ce95640deabd3a70173b062a18f843efad97f5e894c3b01"}
Nov 03 09:07:40.128397 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:40.128383    4473 scope.go:122] "RemoveContainer" containerID="a3d29a7c4944cc360ce95640deabd3a70173b062a18f843efad97f5e894c3b01"
Nov 03 09:07:40.129880 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:40.129853    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:07:40.129985 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:40.129920    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:07:40.130034 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:40.129986    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:07:41.147810 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:41.147774    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w" event={"ID":"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5","Type":"ContainerStarted","Data":"086e7f58b8c853aa1017057f6e20f4082bc9f3ef4d0f05d28356a516af58eb57"}
Nov 03 09:07:41.841193 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:41.841157    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"
Nov 03 09:07:41.842152 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:41.842119    4473 patch_prober.go:29] interesting pod/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 03 09:07:41.842285 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:41.842168    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w" podUID="4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 03 09:07:42.841799 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:42.841759    4473 patch_prober.go:29] interesting pod/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 03 09:07:42.842353 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:42.841826    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w" podUID="4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 03 09:07:43.010479 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:43.010429    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"
Nov 03 09:07:43.841312 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:43.841271    4473 patch_prober.go:29] interesting pod/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 03 09:07:43.841502 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:43.841338    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w" podUID="4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 03 09:07:44.838310 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:44.838272    4473 patch_prober.go:29] interesting pod/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 03 09:07:44.838887 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:44.838855    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w" podUID="4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 03 09:07:45.840972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:45.839323    4473 patch_prober.go:29] interesting pod/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 03 09:07:45.840972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:45.839387    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w" podUID="4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 03 09:07:46.842730 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:46.842694    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"
Nov 03 09:07:46.843581 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:46.843559    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"
Nov 03 09:07:46.937605 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:07:46.937570    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:07:47.069266 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:47.069239    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d06c8459-2mgg9"]
Nov 03 09:07:47.072001 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:47.071977    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d06c8459-2mgg9"]
Nov 03 09:07:47.870478 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:07:47.870449    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7cd80b7-fa65-416a-9221-c9f101e33757" path="/var/lib/kubelet/pods/a7cd80b7-fa65-416a-9221-c9f101e33757/volumes"
Nov 03 09:08:03.673119 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:03.673085    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"]
Nov 03 09:08:03.673673 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:03.673361    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr" podUID="6e740caf-68ae-4528-8965-a1085396deca" containerName="istio-proxy" containerID="cri-o://57e12e2404ff3e78f53433b36e64c37cba27cc506de5459f2697659cda902595" gracePeriod=30
Nov 03 09:08:04.277045 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:04.277014    4473 patch_prober.go:29] interesting pod/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 09:08:04.277236 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:04.277069    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr" podUID="6e740caf-68ae-4528-8965-a1085396deca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 09:08:08.947257 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:08.947197    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"
Nov 03 09:08:09.068121 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.068091    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e740caf-68ae-4528-8965-a1085396deca-istio-token\" (UniqueName: \"kubernetes.io/projected/6e740caf-68ae-4528-8965-a1085396deca-istio-token\") pod \"6e740caf-68ae-4528-8965-a1085396deca\" (UID: \"6e740caf-68ae-4528-8965-a1085396deca\") "
Nov 03 09:08:09.068292 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.068136    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e740caf-68ae-4528-8965-a1085396deca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e740caf-68ae-4528-8965-a1085396deca-workload-certs\") pod \"6e740caf-68ae-4528-8965-a1085396deca\" (UID: \"6e740caf-68ae-4528-8965-a1085396deca\") "
Nov 03 09:08:09.068292 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.068176    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e740caf-68ae-4528-8965-a1085396deca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e740caf-68ae-4528-8965-a1085396deca-credential-socket\") pod \"6e740caf-68ae-4528-8965-a1085396deca\" (UID: \"6e740caf-68ae-4528-8965-a1085396deca\") "
Nov 03 09:08:09.068292 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.068213    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e740caf-68ae-4528-8965-a1085396deca-kube-api-access-b8n42\" (UniqueName: \"kubernetes.io/projected/6e740caf-68ae-4528-8965-a1085396deca-kube-api-access-b8n42\") pod \"6e740caf-68ae-4528-8965-a1085396deca\" (UID: \"6e740caf-68ae-4528-8965-a1085396deca\") "
Nov 03 09:08:09.068292 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.068245    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e740caf-68ae-4528-8965-a1085396deca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e740caf-68ae-4528-8965-a1085396deca-istio-data\") pod \"6e740caf-68ae-4528-8965-a1085396deca\" (UID: \"6e740caf-68ae-4528-8965-a1085396deca\") "
Nov 03 09:08:09.068513 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.068299    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e740caf-68ae-4528-8965-a1085396deca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e740caf-68ae-4528-8965-a1085396deca-istio-podinfo\") pod \"6e740caf-68ae-4528-8965-a1085396deca\" (UID: \"6e740caf-68ae-4528-8965-a1085396deca\") "
Nov 03 09:08:09.068513 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.068371    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e740caf-68ae-4528-8965-a1085396deca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e740caf-68ae-4528-8965-a1085396deca-workload-socket\") pod \"6e740caf-68ae-4528-8965-a1085396deca\" (UID: \"6e740caf-68ae-4528-8965-a1085396deca\") "
Nov 03 09:08:09.068513 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.068452    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e740caf-68ae-4528-8965-a1085396deca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e740caf-68ae-4528-8965-a1085396deca-istiod-ca-cert\") pod \"6e740caf-68ae-4528-8965-a1085396deca\" (UID: \"6e740caf-68ae-4528-8965-a1085396deca\") "
Nov 03 09:08:09.068513 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.068460    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e740caf-68ae-4528-8965-a1085396deca-credential-socket" pod "6e740caf-68ae-4528-8965-a1085396deca" (UID: "6e740caf-68ae-4528-8965-a1085396deca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:08:09.068738 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.068520    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e740caf-68ae-4528-8965-a1085396deca-workload-certs" pod "6e740caf-68ae-4528-8965-a1085396deca" (UID: "6e740caf-68ae-4528-8965-a1085396deca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:08:09.068738 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.068624    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e740caf-68ae-4528-8965-a1085396deca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e740caf-68ae-4528-8965-a1085396deca-istio-envoy\") pod \"6e740caf-68ae-4528-8965-a1085396deca\" (UID: \"6e740caf-68ae-4528-8965-a1085396deca\") "
Nov 03 09:08:09.068850 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.068758    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e740caf-68ae-4528-8965-a1085396deca-istiod-ca-cert" pod "6e740caf-68ae-4528-8965-a1085396deca" (UID: "6e740caf-68ae-4528-8965-a1085396deca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:08:09.068850 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.068810    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e740caf-68ae-4528-8965-a1085396deca-istio-data" pod "6e740caf-68ae-4528-8965-a1085396deca" (UID: "6e740caf-68ae-4528-8965-a1085396deca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:08:09.069144 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.069024    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e740caf-68ae-4528-8965-a1085396deca-workload-socket" pod "6e740caf-68ae-4528-8965-a1085396deca" (UID: "6e740caf-68ae-4528-8965-a1085396deca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:08:09.069309 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.069289    4473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e740caf-68ae-4528-8965-a1085396deca-workload-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:08:09.069407 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.069315    4473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e740caf-68ae-4528-8965-a1085396deca-credential-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:08:09.069407 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.069329    4473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e740caf-68ae-4528-8965-a1085396deca-istio-data\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:08:09.069407 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.069343    4473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e740caf-68ae-4528-8965-a1085396deca-workload-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:08:09.069407 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.069358    4473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e740caf-68ae-4528-8965-a1085396deca-istiod-ca-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:08:09.071313 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.071274    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e740caf-68ae-4528-8965-a1085396deca-istio-podinfo" pod "6e740caf-68ae-4528-8965-a1085396deca" (UID: "6e740caf-68ae-4528-8965-a1085396deca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 09:08:09.071583 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.071555    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e740caf-68ae-4528-8965-a1085396deca-istio-token" pod "6e740caf-68ae-4528-8965-a1085396deca" (UID: "6e740caf-68ae-4528-8965-a1085396deca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:08:09.071583 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.071572    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e740caf-68ae-4528-8965-a1085396deca-istio-envoy" pod "6e740caf-68ae-4528-8965-a1085396deca" (UID: "6e740caf-68ae-4528-8965-a1085396deca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:08:09.071762 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.071592    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e740caf-68ae-4528-8965-a1085396deca-kube-api-access-b8n42" pod "6e740caf-68ae-4528-8965-a1085396deca" (UID: "6e740caf-68ae-4528-8965-a1085396deca"). InnerVolumeSpecName "kube-api-access-b8n42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:08:09.170903 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.170877    4473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e740caf-68ae-4528-8965-a1085396deca-istio-envoy\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:08:09.170903 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.170905    4473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e740caf-68ae-4528-8965-a1085396deca-istio-token\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:08:09.171101 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.170918    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8n42\" (UniqueName: \"kubernetes.io/projected/6e740caf-68ae-4528-8965-a1085396deca-kube-api-access-b8n42\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:08:09.171101 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.170928    4473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e740caf-68ae-4528-8965-a1085396deca-istio-podinfo\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:08:09.418832 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.418800    4473 generic.go:356] "Generic (PLEG): container finished" podID="6e740caf-68ae-4528-8965-a1085396deca" containerID="57e12e2404ff3e78f53433b36e64c37cba27cc506de5459f2697659cda902595" exitCode=0
Nov 03 09:08:09.419058 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.418855    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr" event={"ID":"6e740caf-68ae-4528-8965-a1085396deca","Type":"ContainerDied","Data":"57e12e2404ff3e78f53433b36e64c37cba27cc506de5459f2697659cda902595"}
Nov 03 09:08:09.419058 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.418855    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"
Nov 03 09:08:09.419058 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.418902    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr" event={"ID":"6e740caf-68ae-4528-8965-a1085396deca","Type":"ContainerDied","Data":"a390bbaf87b6025b59076870f9c78ace9e4243cee12d842bc5241f1ed12fbf7e"}
Nov 03 09:08:09.419058 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.418923    4473 scope.go:122] "RemoveContainer" containerID="57e12e2404ff3e78f53433b36e64c37cba27cc506de5459f2697659cda902595"
Nov 03 09:08:09.429891 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.429875    4473 scope.go:122] "RemoveContainer" containerID="88b7efd12000a2b1277ba153152f5a01be2ab91999f6116d182a19e1545b52c2"
Nov 03 09:08:09.438653 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.438638    4473 scope.go:122] "RemoveContainer" containerID="57e12e2404ff3e78f53433b36e64c37cba27cc506de5459f2697659cda902595"
Nov 03 09:08:09.438912 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:08:09.438895    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57e12e2404ff3e78f53433b36e64c37cba27cc506de5459f2697659cda902595\": container with ID starting with 57e12e2404ff3e78f53433b36e64c37cba27cc506de5459f2697659cda902595 not found: ID does not exist" containerID="57e12e2404ff3e78f53433b36e64c37cba27cc506de5459f2697659cda902595"
Nov 03 09:08:09.438966 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.438922    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57e12e2404ff3e78f53433b36e64c37cba27cc506de5459f2697659cda902595"} err="failed to get container status \"57e12e2404ff3e78f53433b36e64c37cba27cc506de5459f2697659cda902595\": rpc error: code = NotFound desc = could not find container \"57e12e2404ff3e78f53433b36e64c37cba27cc506de5459f2697659cda902595\": container with ID starting with 57e12e2404ff3e78f53433b36e64c37cba27cc506de5459f2697659cda902595 not found: ID does not exist"
Nov 03 09:08:09.439018 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.438966    4473 scope.go:122] "RemoveContainer" containerID="88b7efd12000a2b1277ba153152f5a01be2ab91999f6116d182a19e1545b52c2"
Nov 03 09:08:09.439193 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:08:09.439176    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88b7efd12000a2b1277ba153152f5a01be2ab91999f6116d182a19e1545b52c2\": container with ID starting with 88b7efd12000a2b1277ba153152f5a01be2ab91999f6116d182a19e1545b52c2 not found: ID does not exist" containerID="88b7efd12000a2b1277ba153152f5a01be2ab91999f6116d182a19e1545b52c2"
Nov 03 09:08:09.439228 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.439197    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88b7efd12000a2b1277ba153152f5a01be2ab91999f6116d182a19e1545b52c2"} err="failed to get container status \"88b7efd12000a2b1277ba153152f5a01be2ab91999f6116d182a19e1545b52c2\": rpc error: code = NotFound desc = could not find container \"88b7efd12000a2b1277ba153152f5a01be2ab91999f6116d182a19e1545b52c2\": container with ID starting with 88b7efd12000a2b1277ba153152f5a01be2ab91999f6116d182a19e1545b52c2 not found: ID does not exist"
Nov 03 09:08:09.591053 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.591024    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"]
Nov 03 09:08:09.641271 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.641247    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b8fd34-openshift-default-84656d846f-htjzr"]
Nov 03 09:08:09.871117 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:09.871092    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e740caf-68ae-4528-8965-a1085396deca" path="/var/lib/kubelet/pods/6e740caf-68ae-4528-8965-a1085396deca/volumes"
Nov 03 09:08:10.057284 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:10.057252    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b3cbf19-6szrs"]
Nov 03 09:08:10.061265 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:10.061236    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b3cbf19-6szrs"]
Nov 03 09:08:11.873077 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:11.873038    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26d61cea-a8f4-4856-85f7-f9461b2aaede" path="/var/lib/kubelet/pods/26d61cea-a8f4-4856-85f7-f9461b2aaede/volumes"
Nov 03 09:08:14.005858 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:14.005822    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b0c653f4-fp44f"]
Nov 03 09:08:14.055619 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:14.055573    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b0c653f4-fp44f"]
Nov 03 09:08:15.945693 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:15.945653    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa3aefbe-61d6-476e-9a87-d86fec01ecaf" path="/var/lib/kubelet/pods/fa3aefbe-61d6-476e-9a87-d86fec01ecaf/volumes"
Nov 03 09:08:16.938164 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:08:16.938126    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:08:17.062244 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:17.062213    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:08:26.046041 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:26.045995    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-8566fbf99f-47ltl"]
Nov 03 09:08:26.046672 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:26.046653    4473 cpu_manager.go:414] "containerMap: removing container" podUID="d9b64db0-8da5-4bb5-a1dc-d4962912ad9f" containerName="revoke-keys"
Nov 03 09:08:26.046672 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:26.046674    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9b64db0-8da5-4bb5-a1dc-d4962912ad9f" containerName="revoke-keys"
Nov 03 09:08:26.046865 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:26.046694    4473 cpu_manager.go:414] "containerMap: removing container" podUID="6e740caf-68ae-4528-8965-a1085396deca" containerName="istio-proxy"
Nov 03 09:08:26.046865 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:26.046698    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e740caf-68ae-4528-8965-a1085396deca" containerName="istio-proxy"
Nov 03 09:08:26.046865 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:26.046711    4473 cpu_manager.go:414] "containerMap: removing container" podUID="6e740caf-68ae-4528-8965-a1085396deca" containerName="istio-proxy"
Nov 03 09:08:26.046865 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:26.046717    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e740caf-68ae-4528-8965-a1085396deca" containerName="istio-proxy"
Nov 03 09:08:26.046865 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:26.046827    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e740caf-68ae-4528-8965-a1085396deca" containerName="istio-proxy"
Nov 03 09:08:26.046865 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:26.046837    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e740caf-68ae-4528-8965-a1085396deca" containerName="istio-proxy"
Nov 03 09:08:26.046865 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:26.046851    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9b64db0-8da5-4bb5-a1dc-d4962912ad9f" containerName="revoke-keys"
Nov 03 09:08:26.052553 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:26.052530    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-8566fbf99f-47ltl"]
Nov 03 09:08:26.052744 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:26.052720    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-8566fbf99f-47ltl"
Nov 03 09:08:26.157749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:26.157723    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4ae3db-dockercfg-56cj2\""
Nov 03 09:08:26.157890 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:26.157776    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4ae3db\""
Nov 03 09:08:26.234183 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:26.234153    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h27d\" (UniqueName: \"kubernetes.io/projected/ba3d1e4b-5057-429c-9c89-0045b964f1df-kube-api-access-5h27d\") pod \"payload-pre-processing-e2e-ipp-a-4ae3db-8566fbf99f-47ltl\" (UID: \"ba3d1e4b-5057-429c-9c89-0045b964f1df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-8566fbf99f-47ltl"
Nov 03 09:08:26.234330 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:26.234299    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba3d1e4b-5057-429c-9c89-0045b964f1df-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4ae3db-8566fbf99f-47ltl\" (UID: \"ba3d1e4b-5057-429c-9c89-0045b964f1df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-8566fbf99f-47ltl"
Nov 03 09:08:26.335574 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:26.335501    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba3d1e4b-5057-429c-9c89-0045b964f1df-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4ae3db-8566fbf99f-47ltl\" (UID: \"ba3d1e4b-5057-429c-9c89-0045b964f1df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-8566fbf99f-47ltl"
Nov 03 09:08:26.335711 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:26.335636    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h27d\" (UniqueName: \"kubernetes.io/projected/ba3d1e4b-5057-429c-9c89-0045b964f1df-kube-api-access-5h27d\") pod \"payload-pre-processing-e2e-ipp-a-4ae3db-8566fbf99f-47ltl\" (UID: \"ba3d1e4b-5057-429c-9c89-0045b964f1df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-8566fbf99f-47ltl"
Nov 03 09:08:26.336219 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:26.336201    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba3d1e4b-5057-429c-9c89-0045b964f1df-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4ae3db-8566fbf99f-47ltl\" (UID: \"ba3d1e4b-5057-429c-9c89-0045b964f1df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-8566fbf99f-47ltl"
Nov 03 09:08:26.394834 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:26.394800    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h27d\" (UniqueName: \"kubernetes.io/projected/ba3d1e4b-5057-429c-9c89-0045b964f1df-kube-api-access-5h27d\") pod \"payload-pre-processing-e2e-ipp-a-4ae3db-8566fbf99f-47ltl\" (UID: \"ba3d1e4b-5057-429c-9c89-0045b964f1df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-8566fbf99f-47ltl"
Nov 03 09:08:26.668752 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:26.668723    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-8566fbf99f-47ltl"
Nov 03 09:08:27.079655 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:27.079618    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-8566fbf99f-47ltl"]
Nov 03 09:08:27.096143 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:08:27.096108    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba3d1e4b_5057_429c_9c89_0045b964f1df.slice/crio-7de5a670f4460ff05870ac445ce89f81038be4387342367372f603ffdfdcdca5 WatchSource:0}: Error finding container 7de5a670f4460ff05870ac445ce89f81038be4387342367372f603ffdfdcdca5: Status 404 returned error can't find the container with id 7de5a670f4460ff05870ac445ce89f81038be4387342367372f603ffdfdcdca5
Nov 03 09:08:27.588769 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:27.588733    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-8566fbf99f-47ltl" event={"ID":"ba3d1e4b-5057-429c-9c89-0045b964f1df","Type":"ContainerStarted","Data":"7de5a670f4460ff05870ac445ce89f81038be4387342367372f603ffdfdcdca5"}
Nov 03 09:08:28.597257 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:28.597224    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-8566fbf99f-47ltl" event={"ID":"ba3d1e4b-5057-429c-9c89-0045b964f1df","Type":"ContainerStarted","Data":"1503917794ce0d39cb136574badc6828973fa0dbbf2d8e8054383bd1cb93b564"}
Nov 03 09:08:28.597679 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:28.597359    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-8566fbf99f-47ltl"
Nov 03 09:08:28.716298 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:28.716248    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-8566fbf99f-47ltl" podStartSLOduration=3.716234071 podStartE2EDuration="3.716234071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:08:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:08:28.713354586 +0000 UTC m=+1732.982418021" watchObservedRunningTime="2026-11-03 09:08:28.716234071 +0000 UTC m=+1732.985297504"
Nov 03 09:08:30.330916 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:30.330884    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"]
Nov 03 09:08:30.331402 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:30.331167    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6" podUID="5f53106b-dc09-41eb-ab1a-16cd3fa8de90" containerName="istio-proxy" containerID="cri-o://d5ca5ebee09815d7292df4bb424dc78fefd86541767e96f9166851329f93f736" gracePeriod=30
Nov 03 09:08:35.919335 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:35.919289    4473 generic.go:356] "Generic (PLEG): container finished" podID="5f53106b-dc09-41eb-ab1a-16cd3fa8de90" containerID="d5ca5ebee09815d7292df4bb424dc78fefd86541767e96f9166851329f93f736" exitCode=0
Nov 03 09:08:36.061990 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:36.061930    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6" event={"ID":"5f53106b-dc09-41eb-ab1a-16cd3fa8de90","Type":"ContainerDied","Data":"d5ca5ebee09815d7292df4bb424dc78fefd86541767e96f9166851329f93f736"}
Nov 03 09:08:36.078238 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:36.078212    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"
Nov 03 09:08:36.149240 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:36.149209    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-workload-certs\") pod \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\" (UID: \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\") "
Nov 03 09:08:36.149547 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:36.149515    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-workload-certs" pod "5f53106b-dc09-41eb-ab1a-16cd3fa8de90" (UID: "5f53106b-dc09-41eb-ab1a-16cd3fa8de90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:08:36.149740 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:36.149722    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-istiod-ca-cert\") pod \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\" (UID: \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\") "
Nov 03 09:08:36.154382 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:36.154323    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-istio-data\") pod \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\" (UID: \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\") "
Nov 03 09:08:36.154382 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:36.154352    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-kube-api-access-sr9x9\" (UniqueName: \"kubernetes.io/projected/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-kube-api-access-sr9x9\") pod \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\" (UID: \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\") "
Nov 03 09:08:36.154540 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:36.154493    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-istio-envoy\") pod \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\" (UID: \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\") "
Nov 03 09:08:36.154614 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:36.154577    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-istio-podinfo\") pod \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\" (UID: \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\") "
Nov 03 09:08:36.154614 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:36.154608    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-credential-socket\") pod \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\" (UID: \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\") "
Nov 03 09:08:36.154708 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:36.154633    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-istio-token\" (UniqueName: \"kubernetes.io/projected/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-istio-token\") pod \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\" (UID: \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\") "
Nov 03 09:08:36.154708 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:36.154676    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-workload-socket\") pod \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\" (UID: \"5f53106b-dc09-41eb-ab1a-16cd3fa8de90\") "
Nov 03 09:08:36.155166 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:36.155145    4473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-workload-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:08:36.158964 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:36.155994    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-istio-data" pod "5f53106b-dc09-41eb-ab1a-16cd3fa8de90" (UID: "5f53106b-dc09-41eb-ab1a-16cd3fa8de90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:08:36.158964 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:36.156194    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-istiod-ca-cert" pod "5f53106b-dc09-41eb-ab1a-16cd3fa8de90" (UID: "5f53106b-dc09-41eb-ab1a-16cd3fa8de90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:08:36.158964 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:36.156862    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-credential-socket" pod "5f53106b-dc09-41eb-ab1a-16cd3fa8de90" (UID: "5f53106b-dc09-41eb-ab1a-16cd3fa8de90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:08:36.158964 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:36.157676    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-workload-socket" pod "5f53106b-dc09-41eb-ab1a-16cd3fa8de90" (UID: "5f53106b-dc09-41eb-ab1a-16cd3fa8de90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:08:36.164593 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:36.164547    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-istio-token" pod "5f53106b-dc09-41eb-ab1a-16cd3fa8de90" (UID: "5f53106b-dc09-41eb-ab1a-16cd3fa8de90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:08:36.165103 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:36.165066    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-istio-envoy" pod "5f53106b-dc09-41eb-ab1a-16cd3fa8de90" (UID: "5f53106b-dc09-41eb-ab1a-16cd3fa8de90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:08:36.165378 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:36.165343    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-kube-api-access-sr9x9" pod "5f53106b-dc09-41eb-ab1a-16cd3fa8de90" (UID: "5f53106b-dc09-41eb-ab1a-16cd3fa8de90"). InnerVolumeSpecName "kube-api-access-sr9x9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:08:36.168223 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:36.168180    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-istio-podinfo" pod "5f53106b-dc09-41eb-ab1a-16cd3fa8de90" (UID: "5f53106b-dc09-41eb-ab1a-16cd3fa8de90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 09:08:36.257279 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:36.257084    4473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-istio-envoy\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:08:36.257279 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:36.257166    4473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-istio-podinfo\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:08:36.257279 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:36.257183    4473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-credential-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:08:36.257279 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:36.257196    4473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-istio-token\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:08:36.257279 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:36.257210    4473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-workload-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:08:36.257279 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:36.257224    4473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-istiod-ca-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:08:36.257279 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:36.257236    4473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-istio-data\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:08:36.257279 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:36.257250    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr9x9\" (UniqueName: \"kubernetes.io/projected/5f53106b-dc09-41eb-ab1a-16cd3fa8de90-kube-api-access-sr9x9\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:08:36.722015 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:36.695038    4473 scope.go:122] "RemoveContainer" containerID="fb0440613a69cc29295046b4c80b90c0d0f7b01e4c938db843f1b1ad6c0547b1"
Nov 03 09:08:36.897981 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:36.877519    4473 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 09:08:36.897981 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:36.877578    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6" podUID="5f53106b-dc09-41eb-ab1a-16cd3fa8de90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:08:36.949698 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:36.949662    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6" event={"ID":"5f53106b-dc09-41eb-ab1a-16cd3fa8de90","Type":"ContainerDied","Data":"8eeec49b1686c744bd4541d304c64c359cf26768753928a3cf07b714b54ced88"}
Nov 03 09:08:36.951019 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:36.950995    4473 scope.go:122] "RemoveContainer" containerID="d5ca5ebee09815d7292df4bb424dc78fefd86541767e96f9166851329f93f736"
Nov 03 09:08:36.951389 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:36.951355    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"
Nov 03 09:08:37.025058 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:37.025029    4473 scope.go:122] "RemoveContainer" containerID="7f5a3db50f449d4ccfa73ff2f291294ccd1eade95b2c61a634902c23b195dc13"
Nov 03 09:08:37.054037 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:37.053974    4473 scope.go:122] "RemoveContainer" containerID="d5ca5ebee09815d7292df4bb424dc78fefd86541767e96f9166851329f93f736"
Nov 03 09:08:37.054270 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:08:37.054233    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5ca5ebee09815d7292df4bb424dc78fefd86541767e96f9166851329f93f736\": container with ID starting with d5ca5ebee09815d7292df4bb424dc78fefd86541767e96f9166851329f93f736 not found: ID does not exist" containerID="d5ca5ebee09815d7292df4bb424dc78fefd86541767e96f9166851329f93f736"
Nov 03 09:08:37.054377 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:08:37.054280    4473 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"d5ca5ebee09815d7292df4bb424dc78fefd86541767e96f9166851329f93f736\": rpc error: code = NotFound desc = could not find container \"d5ca5ebee09815d7292df4bb424dc78fefd86541767e96f9166851329f93f736\": container with ID starting with d5ca5ebee09815d7292df4bb424dc78fefd86541767e96f9166851329f93f736 not found: ID does not exist" containerID="d5ca5ebee09815d7292df4bb424dc78fefd86541767e96f9166851329f93f736"
Nov 03 09:08:37.054377 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:37.054299    4473 scope.go:122] "RemoveContainer" containerID="34cc879024d822cd92de15349a0336bc25df1b4730ff66d36b44690d018ec013"
Nov 03 09:08:37.064771 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:37.064751    4473 scope.go:122] "RemoveContainer" containerID="2d6700d493748cb2ca4bfaae2713b57bc81f17dd0d2ec4b90656312e9c3c7381"
Nov 03 09:08:37.074789 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:37.074773    4473 scope.go:122] "RemoveContainer" containerID="f47ae2d0e5cb43a239460b34423bab7a00111f4e8be7a2e02943f4c518c353f3"
Nov 03 09:08:37.170592 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:37.170565    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"]
Nov 03 09:08:37.243832 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:37.243800    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-6vvn6"]
Nov 03 09:08:37.870931 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:37.870899    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f53106b-dc09-41eb-ab1a-16cd3fa8de90" path="/var/lib/kubelet/pods/5f53106b-dc09-41eb-ab1a-16cd3fa8de90/volumes"
Nov 03 09:08:39.607097 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:39.607074    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-8566fbf99f-47ltl"
Nov 03 09:08:41.983428 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:41.983383    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-7cd97b9c75-dhs4m"]
Nov 03 09:08:46.938649 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:08:46.938568    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:08:46.983746 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:46.983670    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-7cd97b9c75-dhs4m" podUID="02d38dda-9627-4c78-8b4e-4257f3771afc" containerName="payload-pre-processing" containerID="cri-o://ad4860276f954c6301d5f7095631d66dfe99c550cbbaf2ed15c37ad4723552fc" gracePeriod=25
Nov 03 09:08:47.108791 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:47.108759    4473 generic.go:356] "Generic (PLEG): container finished" podID="5571ff39-4a42-4390-aab3-2f1e60011a4e" containerID="f7812c462d79f52797cef5c89402e0d737854f743ce110893ab312c840d0e0b1" exitCode=0
Nov 03 09:08:47.108985 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:47.108847    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4" event={"ID":"5571ff39-4a42-4390-aab3-2f1e60011a4e","Type":"ContainerDied","Data":"f7812c462d79f52797cef5c89402e0d737854f743ce110893ab312c840d0e0b1"}
Nov 03 09:08:47.109393 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:47.109371    4473 scope.go:122] "RemoveContainer" containerID="f7812c462d79f52797cef5c89402e0d737854f743ce110893ab312c840d0e0b1"
Nov 03 09:08:47.111288 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:47.111226    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:08:47.111381 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:47.111321    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:08:47.111381 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:47.111356    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:08:47.111469 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:47.111384    4473 generic.go:356] "Generic (PLEG): container finished" podID="02d38dda-9627-4c78-8b4e-4257f3771afc" containerID="ad4860276f954c6301d5f7095631d66dfe99c550cbbaf2ed15c37ad4723552fc" exitCode=0
Nov 03 09:08:47.111525 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:47.111507    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-7cd97b9c75-dhs4m" event={"ID":"02d38dda-9627-4c78-8b4e-4257f3771afc","Type":"ContainerDied","Data":"ad4860276f954c6301d5f7095631d66dfe99c550cbbaf2ed15c37ad4723552fc"}
Nov 03 09:08:47.412516 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:47.412487    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-7cd97b9c75-dhs4m"
Nov 03 09:08:47.558359 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:47.558276    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02d38dda-9627-4c78-8b4e-4257f3771afc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02d38dda-9627-4c78-8b4e-4257f3771afc-plugins-config-volume\") pod \"02d38dda-9627-4c78-8b4e-4257f3771afc\" (UID: \"02d38dda-9627-4c78-8b4e-4257f3771afc\") "
Nov 03 09:08:47.558519 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:47.558399    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02d38dda-9627-4c78-8b4e-4257f3771afc-kube-api-access-wrgsv\" (UniqueName: \"kubernetes.io/projected/02d38dda-9627-4c78-8b4e-4257f3771afc-kube-api-access-wrgsv\") pod \"02d38dda-9627-4c78-8b4e-4257f3771afc\" (UID: \"02d38dda-9627-4c78-8b4e-4257f3771afc\") "
Nov 03 09:08:47.558661 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:47.558634    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02d38dda-9627-4c78-8b4e-4257f3771afc-plugins-config-volume" pod "02d38dda-9627-4c78-8b4e-4257f3771afc" (UID: "02d38dda-9627-4c78-8b4e-4257f3771afc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:08:47.559038 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:47.559020    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02d38dda-9627-4c78-8b4e-4257f3771afc-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:08:47.560662 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:47.560642    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02d38dda-9627-4c78-8b4e-4257f3771afc-kube-api-access-wrgsv" pod "02d38dda-9627-4c78-8b4e-4257f3771afc" (UID: "02d38dda-9627-4c78-8b4e-4257f3771afc"). InnerVolumeSpecName "kube-api-access-wrgsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:08:47.660095 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:47.660063    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrgsv\" (UniqueName: \"kubernetes.io/projected/02d38dda-9627-4c78-8b4e-4257f3771afc-kube-api-access-wrgsv\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:08:48.164369 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:48.162927    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-7cd97b9c75-dhs4m" event={"ID":"02d38dda-9627-4c78-8b4e-4257f3771afc","Type":"ContainerDied","Data":"4e5115f9d53cb3537903cb0d15291dbdbc7a024867674972fe0e134b3fbb2cd2"}
Nov 03 09:08:48.164369 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:48.163034    4473 scope.go:122] "RemoveContainer" containerID="ad4860276f954c6301d5f7095631d66dfe99c550cbbaf2ed15c37ad4723552fc"
Nov 03 09:08:48.164369 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:48.163235    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-7cd97b9c75-dhs4m"
Nov 03 09:08:48.242988 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:48.228138    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4" event={"ID":"5571ff39-4a42-4390-aab3-2f1e60011a4e","Type":"ContainerStarted","Data":"e14b205c816fe103644b4fe18af95cd706fad49f77066b724dbd37700c997ff6"}
Nov 03 09:08:48.275967 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:48.272465    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"
Nov 03 09:08:48.275967 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:48.272598    4473 patch_prober.go:29] interesting pod/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 03 09:08:48.275967 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:48.272661    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4" podUID="5571ff39-4a42-4390-aab3-2f1e60011a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 03 09:08:48.490094 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:48.490007    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-7cd97b9c75-dhs4m"]
Nov 03 09:08:48.495437 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:48.495408    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-04934e-7cd97b9c75-dhs4m"]
Nov 03 09:08:49.270741 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:49.270707    4473 patch_prober.go:29] interesting pod/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 03 09:08:49.271213 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:49.270772    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4" podUID="5571ff39-4a42-4390-aab3-2f1e60011a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 03 09:08:49.871451 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:49.871419    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02d38dda-9627-4c78-8b4e-4257f3771afc" path="/var/lib/kubelet/pods/02d38dda-9627-4c78-8b4e-4257f3771afc/volumes"
Nov 03 09:08:50.270334 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:50.270257    4473 patch_prober.go:29] interesting pod/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 03 09:08:50.270334 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:50.270325    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4" podUID="5571ff39-4a42-4390-aab3-2f1e60011a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 03 09:08:50.372478 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:50.372438    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"
Nov 03 09:08:51.344970 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:51.344341    4473 patch_prober.go:29] interesting pod/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 03 09:08:51.344970 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:51.344421    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4" podUID="5571ff39-4a42-4390-aab3-2f1e60011a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 03 09:08:52.270299 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:52.270252    4473 patch_prober.go:29] interesting pod/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 03 09:08:52.270725 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:52.270330    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4" podUID="5571ff39-4a42-4390-aab3-2f1e60011a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 03 09:08:53.270607 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:53.270571    4473 patch_prober.go:29] interesting pod/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Nov 03 09:08:53.271064 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:53.270636    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4" podUID="5571ff39-4a42-4390-aab3-2f1e60011a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Nov 03 09:08:53.473971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:53.464418    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"]
Nov 03 09:08:53.473971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:53.464654    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68" podUID="625c61e9-f360-4f60-9b21-885c4696aba3" containerName="istio-proxy" containerID="cri-o://90d3b4e38efc4635aaa4742962ac9f70cd966ca053357c1cf00b6a4949c8f2e4" gracePeriod=30
Nov 03 09:08:54.274709 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:54.274682    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"
Nov 03 09:08:54.275989 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:54.275967    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"
Nov 03 09:08:58.738010 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:58.737935    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:08:58.798509 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:58.798483    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/625c61e9-f360-4f60-9b21-885c4696aba3-kube-api-access-t4rpk\" (UniqueName: \"kubernetes.io/projected/625c61e9-f360-4f60-9b21-885c4696aba3-kube-api-access-t4rpk\") pod \"625c61e9-f360-4f60-9b21-885c4696aba3\" (UID: \"625c61e9-f360-4f60-9b21-885c4696aba3\") "
Nov 03 09:08:58.798660 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:58.798518    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/625c61e9-f360-4f60-9b21-885c4696aba3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/625c61e9-f360-4f60-9b21-885c4696aba3-istio-envoy\") pod \"625c61e9-f360-4f60-9b21-885c4696aba3\" (UID: \"625c61e9-f360-4f60-9b21-885c4696aba3\") "
Nov 03 09:08:58.798660 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:58.798546    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/625c61e9-f360-4f60-9b21-885c4696aba3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/625c61e9-f360-4f60-9b21-885c4696aba3-istio-data\") pod \"625c61e9-f360-4f60-9b21-885c4696aba3\" (UID: \"625c61e9-f360-4f60-9b21-885c4696aba3\") "
Nov 03 09:08:58.798660 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:58.798605    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/625c61e9-f360-4f60-9b21-885c4696aba3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/625c61e9-f360-4f60-9b21-885c4696aba3-istio-podinfo\") pod \"625c61e9-f360-4f60-9b21-885c4696aba3\" (UID: \"625c61e9-f360-4f60-9b21-885c4696aba3\") "
Nov 03 09:08:58.798793 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:58.798685    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/625c61e9-f360-4f60-9b21-885c4696aba3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/625c61e9-f360-4f60-9b21-885c4696aba3-workload-socket\") pod \"625c61e9-f360-4f60-9b21-885c4696aba3\" (UID: \"625c61e9-f360-4f60-9b21-885c4696aba3\") "
Nov 03 09:08:58.798793 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:58.798707    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/625c61e9-f360-4f60-9b21-885c4696aba3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/625c61e9-f360-4f60-9b21-885c4696aba3-credential-socket\") pod \"625c61e9-f360-4f60-9b21-885c4696aba3\" (UID: \"625c61e9-f360-4f60-9b21-885c4696aba3\") "
Nov 03 09:08:58.798793 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:58.798745    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/625c61e9-f360-4f60-9b21-885c4696aba3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/625c61e9-f360-4f60-9b21-885c4696aba3-workload-certs\") pod \"625c61e9-f360-4f60-9b21-885c4696aba3\" (UID: \"625c61e9-f360-4f60-9b21-885c4696aba3\") "
Nov 03 09:08:58.798999 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:58.798865    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/625c61e9-f360-4f60-9b21-885c4696aba3-istio-token\" (UniqueName: \"kubernetes.io/projected/625c61e9-f360-4f60-9b21-885c4696aba3-istio-token\") pod \"625c61e9-f360-4f60-9b21-885c4696aba3\" (UID: \"625c61e9-f360-4f60-9b21-885c4696aba3\") "
Nov 03 09:08:58.798999 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:58.798880    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/625c61e9-f360-4f60-9b21-885c4696aba3-istio-data" pod "625c61e9-f360-4f60-9b21-885c4696aba3" (UID: "625c61e9-f360-4f60-9b21-885c4696aba3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:08:58.798999 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:58.798920    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/625c61e9-f360-4f60-9b21-885c4696aba3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/625c61e9-f360-4f60-9b21-885c4696aba3-istiod-ca-cert\") pod \"625c61e9-f360-4f60-9b21-885c4696aba3\" (UID: \"625c61e9-f360-4f60-9b21-885c4696aba3\") "
Nov 03 09:08:58.798999 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:58.798987    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/625c61e9-f360-4f60-9b21-885c4696aba3-workload-socket" pod "625c61e9-f360-4f60-9b21-885c4696aba3" (UID: "625c61e9-f360-4f60-9b21-885c4696aba3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:08:58.799250 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:58.799112    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/625c61e9-f360-4f60-9b21-885c4696aba3-credential-socket" pod "625c61e9-f360-4f60-9b21-885c4696aba3" (UID: "625c61e9-f360-4f60-9b21-885c4696aba3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:08:58.799608 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:58.799583    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/625c61e9-f360-4f60-9b21-885c4696aba3-workload-certs" pod "625c61e9-f360-4f60-9b21-885c4696aba3" (UID: "625c61e9-f360-4f60-9b21-885c4696aba3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:08:58.799877 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:58.799688    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/625c61e9-f360-4f60-9b21-885c4696aba3-istiod-ca-cert" pod "625c61e9-f360-4f60-9b21-885c4696aba3" (UID: "625c61e9-f360-4f60-9b21-885c4696aba3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:08:58.799877 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:58.799767    4473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/625c61e9-f360-4f60-9b21-885c4696aba3-istio-data\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:08:58.799877 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:58.799789    4473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/625c61e9-f360-4f60-9b21-885c4696aba3-workload-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:08:58.799877 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:58.799805    4473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/625c61e9-f360-4f60-9b21-885c4696aba3-credential-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:08:58.801039 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:58.801013    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/625c61e9-f360-4f60-9b21-885c4696aba3-kube-api-access-t4rpk" pod "625c61e9-f360-4f60-9b21-885c4696aba3" (UID: "625c61e9-f360-4f60-9b21-885c4696aba3"). InnerVolumeSpecName "kube-api-access-t4rpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:08:58.801113 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:58.801086    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/625c61e9-f360-4f60-9b21-885c4696aba3-istio-envoy" pod "625c61e9-f360-4f60-9b21-885c4696aba3" (UID: "625c61e9-f360-4f60-9b21-885c4696aba3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:08:58.801204 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:58.801186    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/625c61e9-f360-4f60-9b21-885c4696aba3-istio-token" pod "625c61e9-f360-4f60-9b21-885c4696aba3" (UID: "625c61e9-f360-4f60-9b21-885c4696aba3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:08:58.801314 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:58.801296    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/625c61e9-f360-4f60-9b21-885c4696aba3-istio-podinfo" pod "625c61e9-f360-4f60-9b21-885c4696aba3" (UID: "625c61e9-f360-4f60-9b21-885c4696aba3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 09:08:58.901742 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:58.901711    4473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/625c61e9-f360-4f60-9b21-885c4696aba3-istio-token\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:08:58.901742 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:58.901744    4473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/625c61e9-f360-4f60-9b21-885c4696aba3-istiod-ca-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:08:58.902014 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:58.901762    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4rpk\" (UniqueName: \"kubernetes.io/projected/625c61e9-f360-4f60-9b21-885c4696aba3-kube-api-access-t4rpk\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:08:58.902014 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:58.901777    4473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/625c61e9-f360-4f60-9b21-885c4696aba3-istio-envoy\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:08:58.902014 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:58.901792    4473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/625c61e9-f360-4f60-9b21-885c4696aba3-istio-podinfo\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:08:58.902014 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:58.901807    4473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/625c61e9-f360-4f60-9b21-885c4696aba3-workload-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:08:59.052275 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:59.052189    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc48c856-z8zwb"]
Nov 03 09:08:59.060292 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:59.060271    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc48c856-z8zwb"]
Nov 03 09:08:59.426922 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:59.426889    4473 generic.go:356] "Generic (PLEG): container finished" podID="625c61e9-f360-4f60-9b21-885c4696aba3" containerID="90d3b4e38efc4635aaa4742962ac9f70cd966ca053357c1cf00b6a4949c8f2e4" exitCode=0
Nov 03 09:08:59.427154 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:59.426958    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"
Nov 03 09:08:59.427154 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:59.426977    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68" event={"ID":"625c61e9-f360-4f60-9b21-885c4696aba3","Type":"ContainerDied","Data":"90d3b4e38efc4635aaa4742962ac9f70cd966ca053357c1cf00b6a4949c8f2e4"}
Nov 03 09:08:59.427154 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:59.427032    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68" event={"ID":"625c61e9-f360-4f60-9b21-885c4696aba3","Type":"ContainerDied","Data":"1837a60ac251b2df674fbf46ee7c2a579ea3c0f74b80429f5ca258891f1d0d57"}
Nov 03 09:08:59.427154 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:59.427054    4473 scope.go:122] "RemoveContainer" containerID="90d3b4e38efc4635aaa4742962ac9f70cd966ca053357c1cf00b6a4949c8f2e4"
Nov 03 09:08:59.437500 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:59.437485    4473 scope.go:122] "RemoveContainer" containerID="8e2707fcf56521b5ddf62f7183fd97098e64a10cd344cb52fecbaf9ac901a00e"
Nov 03 09:08:59.447086 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:59.447067    4473 scope.go:122] "RemoveContainer" containerID="90d3b4e38efc4635aaa4742962ac9f70cd966ca053357c1cf00b6a4949c8f2e4"
Nov 03 09:08:59.447331 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:08:59.447315    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90d3b4e38efc4635aaa4742962ac9f70cd966ca053357c1cf00b6a4949c8f2e4\": container with ID starting with 90d3b4e38efc4635aaa4742962ac9f70cd966ca053357c1cf00b6a4949c8f2e4 not found: ID does not exist" containerID="90d3b4e38efc4635aaa4742962ac9f70cd966ca053357c1cf00b6a4949c8f2e4"
Nov 03 09:08:59.447389 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:59.447339    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90d3b4e38efc4635aaa4742962ac9f70cd966ca053357c1cf00b6a4949c8f2e4"} err="failed to get container status \"90d3b4e38efc4635aaa4742962ac9f70cd966ca053357c1cf00b6a4949c8f2e4\": rpc error: code = NotFound desc = could not find container \"90d3b4e38efc4635aaa4742962ac9f70cd966ca053357c1cf00b6a4949c8f2e4\": container with ID starting with 90d3b4e38efc4635aaa4742962ac9f70cd966ca053357c1cf00b6a4949c8f2e4 not found: ID does not exist"
Nov 03 09:08:59.447389 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:59.447357    4473 scope.go:122] "RemoveContainer" containerID="8e2707fcf56521b5ddf62f7183fd97098e64a10cd344cb52fecbaf9ac901a00e"
Nov 03 09:08:59.447613 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:08:59.447597    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e2707fcf56521b5ddf62f7183fd97098e64a10cd344cb52fecbaf9ac901a00e\": container with ID starting with 8e2707fcf56521b5ddf62f7183fd97098e64a10cd344cb52fecbaf9ac901a00e not found: ID does not exist" containerID="8e2707fcf56521b5ddf62f7183fd97098e64a10cd344cb52fecbaf9ac901a00e"
Nov 03 09:08:59.447661 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:59.447619    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e2707fcf56521b5ddf62f7183fd97098e64a10cd344cb52fecbaf9ac901a00e"} err="failed to get container status \"8e2707fcf56521b5ddf62f7183fd97098e64a10cd344cb52fecbaf9ac901a00e\": rpc error: code = NotFound desc = could not find container \"8e2707fcf56521b5ddf62f7183fd97098e64a10cd344cb52fecbaf9ac901a00e\": container with ID starting with 8e2707fcf56521b5ddf62f7183fd97098e64a10cd344cb52fecbaf9ac901a00e not found: ID does not exist"
Nov 03 09:08:59.598542 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:59.598506    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"]
Nov 03 09:08:59.649433 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:59.649395    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-204dfc-openshift-default-65bb849dc9-wxx68"]
Nov 03 09:08:59.870337 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:59.870307    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="625c61e9-f360-4f60-9b21-885c4696aba3" path="/var/lib/kubelet/pods/625c61e9-f360-4f60-9b21-885c4696aba3/volumes"
Nov 03 09:08:59.870765 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:08:59.870751    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76a4c9d3-b074-41ee-a4f9-8ffab40c60e9" path="/var/lib/kubelet/pods/76a4c9d3-b074-41ee-a4f9-8ffab40c60e9/volumes"
Nov 03 09:09:16.942495 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:09:16.942455    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:09:17.017298 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:17.017260    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:09:20.210535 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:20.210501    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"]
Nov 03 09:09:20.211037 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:20.210716    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8" podUID="4ecc8182-a9fd-471f-a00d-2195f2757dfd" containerName="istio-proxy" containerID="cri-o://661b822d3eac4602afaa31eb245b7404f4f100f635b76ac856c0a0a28a4d4933" gracePeriod=30
Nov 03 09:09:25.490463 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:25.490404    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"
Nov 03 09:09:25.601497 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:25.601467    4473 generic.go:356] "Generic (PLEG): container finished" podID="4ecc8182-a9fd-471f-a00d-2195f2757dfd" containerID="661b822d3eac4602afaa31eb245b7404f4f100f635b76ac856c0a0a28a4d4933" exitCode=0
Nov 03 09:09:25.601682 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:25.601515    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"
Nov 03 09:09:25.601682 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:25.601554    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8" event={"ID":"4ecc8182-a9fd-471f-a00d-2195f2757dfd","Type":"ContainerDied","Data":"661b822d3eac4602afaa31eb245b7404f4f100f635b76ac856c0a0a28a4d4933"}
Nov 03 09:09:25.601682 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:25.601620    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8" event={"ID":"4ecc8182-a9fd-471f-a00d-2195f2757dfd","Type":"ContainerDied","Data":"1243d863bc9b41ab680d877842478c208ec344a7df190ab7ab416a35b6ba9c91"}
Nov 03 09:09:25.601682 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:25.601649    4473 scope.go:122] "RemoveContainer" containerID="661b822d3eac4602afaa31eb245b7404f4f100f635b76ac856c0a0a28a4d4933"
Nov 03 09:09:25.611150 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:25.611126    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ecc8182-a9fd-471f-a00d-2195f2757dfd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ecc8182-a9fd-471f-a00d-2195f2757dfd-istio-data\") pod \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\" (UID: \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\") "
Nov 03 09:09:25.611285 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:25.611234    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ecc8182-a9fd-471f-a00d-2195f2757dfd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ecc8182-a9fd-471f-a00d-2195f2757dfd-istio-podinfo\") pod \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\" (UID: \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\") "
Nov 03 09:09:25.611285 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:25.611274    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ecc8182-a9fd-471f-a00d-2195f2757dfd-istio-token\" (UniqueName: \"kubernetes.io/projected/4ecc8182-a9fd-471f-a00d-2195f2757dfd-istio-token\") pod \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\" (UID: \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\") "
Nov 03 09:09:25.611404 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:25.611311    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ecc8182-a9fd-471f-a00d-2195f2757dfd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ecc8182-a9fd-471f-a00d-2195f2757dfd-istiod-ca-cert\") pod \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\" (UID: \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\") "
Nov 03 09:09:25.611404 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:25.611345    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ecc8182-a9fd-471f-a00d-2195f2757dfd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ecc8182-a9fd-471f-a00d-2195f2757dfd-workload-socket\") pod \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\" (UID: \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\") "
Nov 03 09:09:25.611404 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:25.611382    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ecc8182-a9fd-471f-a00d-2195f2757dfd-istio-data" pod "4ecc8182-a9fd-471f-a00d-2195f2757dfd" (UID: "4ecc8182-a9fd-471f-a00d-2195f2757dfd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:09:25.611559 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:25.611448    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ecc8182-a9fd-471f-a00d-2195f2757dfd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ecc8182-a9fd-471f-a00d-2195f2757dfd-istio-envoy\") pod \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\" (UID: \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\") "
Nov 03 09:09:25.611559 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:25.611507    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ecc8182-a9fd-471f-a00d-2195f2757dfd-kube-api-access-sfj87\" (UniqueName: \"kubernetes.io/projected/4ecc8182-a9fd-471f-a00d-2195f2757dfd-kube-api-access-sfj87\") pod \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\" (UID: \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\") "
Nov 03 09:09:25.611710 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:25.611586    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ecc8182-a9fd-471f-a00d-2195f2757dfd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ecc8182-a9fd-471f-a00d-2195f2757dfd-workload-certs\") pod \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\" (UID: \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\") "
Nov 03 09:09:25.611710 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:25.611672    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ecc8182-a9fd-471f-a00d-2195f2757dfd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ecc8182-a9fd-471f-a00d-2195f2757dfd-credential-socket\") pod \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\" (UID: \"4ecc8182-a9fd-471f-a00d-2195f2757dfd\") "
Nov 03 09:09:25.611983 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:25.611669    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ecc8182-a9fd-471f-a00d-2195f2757dfd-workload-socket" pod "4ecc8182-a9fd-471f-a00d-2195f2757dfd" (UID: "4ecc8182-a9fd-471f-a00d-2195f2757dfd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:09:25.611983 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:25.611727    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ecc8182-a9fd-471f-a00d-2195f2757dfd-istiod-ca-cert" pod "4ecc8182-a9fd-471f-a00d-2195f2757dfd" (UID: "4ecc8182-a9fd-471f-a00d-2195f2757dfd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:09:25.612193 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:25.611985    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ecc8182-a9fd-471f-a00d-2195f2757dfd-credential-socket" pod "4ecc8182-a9fd-471f-a00d-2195f2757dfd" (UID: "4ecc8182-a9fd-471f-a00d-2195f2757dfd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:09:25.612193 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:25.612060    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ecc8182-a9fd-471f-a00d-2195f2757dfd-workload-certs" pod "4ecc8182-a9fd-471f-a00d-2195f2757dfd" (UID: "4ecc8182-a9fd-471f-a00d-2195f2757dfd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:09:25.612193 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:25.612155    4473 scope.go:122] "RemoveContainer" containerID="661b822d3eac4602afaa31eb245b7404f4f100f635b76ac856c0a0a28a4d4933"
Nov 03 09:09:25.612361 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:25.612324    4473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ecc8182-a9fd-471f-a00d-2195f2757dfd-istio-data\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:09:25.612361 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:25.612347    4473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ecc8182-a9fd-471f-a00d-2195f2757dfd-istiod-ca-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:09:25.612470 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:25.612363    4473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ecc8182-a9fd-471f-a00d-2195f2757dfd-workload-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:09:25.612470 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:25.612378    4473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ecc8182-a9fd-471f-a00d-2195f2757dfd-workload-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:09:25.612470 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:25.612393    4473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ecc8182-a9fd-471f-a00d-2195f2757dfd-credential-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:09:25.612470 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:09:25.612444    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"661b822d3eac4602afaa31eb245b7404f4f100f635b76ac856c0a0a28a4d4933\": container with ID starting with 661b822d3eac4602afaa31eb245b7404f4f100f635b76ac856c0a0a28a4d4933 not found: ID does not exist" containerID="661b822d3eac4602afaa31eb245b7404f4f100f635b76ac856c0a0a28a4d4933"
Nov 03 09:09:25.612863 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:25.612478    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"661b822d3eac4602afaa31eb245b7404f4f100f635b76ac856c0a0a28a4d4933"} err="failed to get container status \"661b822d3eac4602afaa31eb245b7404f4f100f635b76ac856c0a0a28a4d4933\": rpc error: code = NotFound desc = could not find container \"661b822d3eac4602afaa31eb245b7404f4f100f635b76ac856c0a0a28a4d4933\": container with ID starting with 661b822d3eac4602afaa31eb245b7404f4f100f635b76ac856c0a0a28a4d4933 not found: ID does not exist"
Nov 03 09:09:25.613669 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:25.613644    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ecc8182-a9fd-471f-a00d-2195f2757dfd-istio-podinfo" pod "4ecc8182-a9fd-471f-a00d-2195f2757dfd" (UID: "4ecc8182-a9fd-471f-a00d-2195f2757dfd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 09:09:25.614126 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:25.614102    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ecc8182-a9fd-471f-a00d-2195f2757dfd-istio-envoy" pod "4ecc8182-a9fd-471f-a00d-2195f2757dfd" (UID: "4ecc8182-a9fd-471f-a00d-2195f2757dfd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:09:25.614290 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:25.614269    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ecc8182-a9fd-471f-a00d-2195f2757dfd-istio-token" pod "4ecc8182-a9fd-471f-a00d-2195f2757dfd" (UID: "4ecc8182-a9fd-471f-a00d-2195f2757dfd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:09:25.614421 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:25.614398    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ecc8182-a9fd-471f-a00d-2195f2757dfd-kube-api-access-sfj87" pod "4ecc8182-a9fd-471f-a00d-2195f2757dfd" (UID: "4ecc8182-a9fd-471f-a00d-2195f2757dfd"). InnerVolumeSpecName "kube-api-access-sfj87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:09:25.713626 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:25.713588    4473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ecc8182-a9fd-471f-a00d-2195f2757dfd-istio-podinfo\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:09:25.713626 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:25.713617    4473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ecc8182-a9fd-471f-a00d-2195f2757dfd-istio-token\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:09:25.713626 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:25.713627    4473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ecc8182-a9fd-471f-a00d-2195f2757dfd-istio-envoy\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:09:25.713850 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:25.713636    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfj87\" (UniqueName: \"kubernetes.io/projected/4ecc8182-a9fd-471f-a00d-2195f2757dfd-kube-api-access-sfj87\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:09:26.020327 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:26.020289    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"]
Nov 03 09:09:26.026023 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:26.025997    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-nr5n8"]
Nov 03 09:09:27.870992 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:27.870933    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ecc8182-a9fd-471f-a00d-2195f2757dfd" path="/var/lib/kubelet/pods/4ecc8182-a9fd-471f-a00d-2195f2757dfd/volumes"
Nov 03 09:09:36.262116 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:36.262091    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 09:09:36.264516 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:36.264492    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 09:09:37.146551 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:37.146524    4473 scope.go:122] "RemoveContainer" containerID="3eb952460cf10f107496d45b2f1c9511afb683d039d653f2dd23a489adf9c133"
Nov 03 09:09:37.156120 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:37.156102    4473 scope.go:122] "RemoveContainer" containerID="1159eac4f000d90cf6344eb0e9e010a6d2fc33881fb72b850918e15e43030d92"
Nov 03 09:09:37.520045 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:37.519939    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-04934e-548676699b-sctv8"]
Nov 03 09:09:37.520696 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:37.520664    4473 cpu_manager.go:414] "containerMap: removing container" podUID="625c61e9-f360-4f60-9b21-885c4696aba3" containerName="istio-proxy"
Nov 03 09:09:37.520796 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:37.520696    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="625c61e9-f360-4f60-9b21-885c4696aba3" containerName="istio-proxy"
Nov 03 09:09:37.520796 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:37.520715    4473 cpu_manager.go:414] "containerMap: removing container" podUID="625c61e9-f360-4f60-9b21-885c4696aba3" containerName="istio-proxy"
Nov 03 09:09:37.520796 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:37.520722    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="625c61e9-f360-4f60-9b21-885c4696aba3" containerName="istio-proxy"
Nov 03 09:09:37.520796 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:37.520759    4473 cpu_manager.go:414] "containerMap: removing container" podUID="4ecc8182-a9fd-471f-a00d-2195f2757dfd" containerName="istio-proxy"
Nov 03 09:09:37.520796 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:37.520766    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ecc8182-a9fd-471f-a00d-2195f2757dfd" containerName="istio-proxy"
Nov 03 09:09:37.521085 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:37.520798    4473 cpu_manager.go:414] "containerMap: removing container" podUID="5f53106b-dc09-41eb-ab1a-16cd3fa8de90" containerName="istio-proxy"
Nov 03 09:09:37.521085 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:37.520806    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f53106b-dc09-41eb-ab1a-16cd3fa8de90" containerName="istio-proxy"
Nov 03 09:09:37.521085 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:37.520833    4473 cpu_manager.go:414] "containerMap: removing container" podUID="02d38dda-9627-4c78-8b4e-4257f3771afc" containerName="payload-pre-processing"
Nov 03 09:09:37.521085 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:37.520840    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02d38dda-9627-4c78-8b4e-4257f3771afc" containerName="payload-pre-processing"
Nov 03 09:09:37.521085 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:37.521008    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f53106b-dc09-41eb-ab1a-16cd3fa8de90" containerName="istio-proxy"
Nov 03 09:09:37.521085 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:37.521028    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="02d38dda-9627-4c78-8b4e-4257f3771afc" containerName="payload-pre-processing"
Nov 03 09:09:37.521085 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:37.521041    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="625c61e9-f360-4f60-9b21-885c4696aba3" containerName="istio-proxy"
Nov 03 09:09:37.521085 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:37.521056    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="625c61e9-f360-4f60-9b21-885c4696aba3" containerName="istio-proxy"
Nov 03 09:09:37.521085 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:37.521070    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ecc8182-a9fd-471f-a00d-2195f2757dfd" containerName="istio-proxy"
Nov 03 09:09:37.526201 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:37.526179    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-04934e-548676699b-sctv8"
Nov 03 09:09:37.546767 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:37.546739    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnd77\" (UniqueName: \"kubernetes.io/projected/d59877f4-7a69-496e-b04b-7c3df4c9a2c4-kube-api-access-pnd77\") pod \"payload-processing-e2e-ipp-a-04934e-548676699b-sctv8\" (UID: \"d59877f4-7a69-496e-b04b-7c3df4c9a2c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-04934e-548676699b-sctv8"
Nov 03 09:09:37.546870 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:37.546792    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d59877f4-7a69-496e-b04b-7c3df4c9a2c4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-04934e-548676699b-sctv8\" (UID: \"d59877f4-7a69-496e-b04b-7c3df4c9a2c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-04934e-548676699b-sctv8"
Nov 03 09:09:37.581243 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:37.581219    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-04934e-548676699b-sctv8"]
Nov 03 09:09:37.628078 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:37.628052    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-04934e\""
Nov 03 09:09:37.628207 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:37.628058    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-04934e-dockercfg-xg4kf\""
Nov 03 09:09:37.647748 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:37.647722    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnd77\" (UniqueName: \"kubernetes.io/projected/d59877f4-7a69-496e-b04b-7c3df4c9a2c4-kube-api-access-pnd77\") pod \"payload-processing-e2e-ipp-a-04934e-548676699b-sctv8\" (UID: \"d59877f4-7a69-496e-b04b-7c3df4c9a2c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-04934e-548676699b-sctv8"
Nov 03 09:09:37.647844 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:37.647768    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d59877f4-7a69-496e-b04b-7c3df4c9a2c4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-04934e-548676699b-sctv8\" (UID: \"d59877f4-7a69-496e-b04b-7c3df4c9a2c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-04934e-548676699b-sctv8"
Nov 03 09:09:37.648435 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:37.648419    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d59877f4-7a69-496e-b04b-7c3df4c9a2c4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-04934e-548676699b-sctv8\" (UID: \"d59877f4-7a69-496e-b04b-7c3df4c9a2c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-04934e-548676699b-sctv8"
Nov 03 09:09:37.707564 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:37.707535    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnd77\" (UniqueName: \"kubernetes.io/projected/d59877f4-7a69-496e-b04b-7c3df4c9a2c4-kube-api-access-pnd77\") pod \"payload-processing-e2e-ipp-a-04934e-548676699b-sctv8\" (UID: \"d59877f4-7a69-496e-b04b-7c3df4c9a2c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-04934e-548676699b-sctv8"
Nov 03 09:09:37.839306 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:37.839227    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-04934e-548676699b-sctv8"
Nov 03 09:09:38.216832 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:38.216799    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-04934e-548676699b-sctv8"]
Nov 03 09:09:38.217200 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:09:38.217178    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd59877f4_7a69_496e_b04b_7c3df4c9a2c4.slice/crio-859a13a17e1b14cc5a044de71a84f5299096273536e62d902f219e55b18f65e1 WatchSource:0}: Error finding container 859a13a17e1b14cc5a044de71a84f5299096273536e62d902f219e55b18f65e1: Status 404 returned error can't find the container with id 859a13a17e1b14cc5a044de71a84f5299096273536e62d902f219e55b18f65e1
Nov 03 09:09:38.681568 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:38.681536    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-04934e-548676699b-sctv8" event={"ID":"d59877f4-7a69-496e-b04b-7c3df4c9a2c4","Type":"ContainerStarted","Data":"859a13a17e1b14cc5a044de71a84f5299096273536e62d902f219e55b18f65e1"}
Nov 03 09:09:40.696360 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:40.696330    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-04934e-548676699b-sctv8" event={"ID":"d59877f4-7a69-496e-b04b-7c3df4c9a2c4","Type":"ContainerStarted","Data":"6a0e54fe21e8e0d65c0b955489b981812406788c38cc82877067734616e76075"}
Nov 03 09:09:40.696782 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:40.696463    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-04934e-548676699b-sctv8"
Nov 03 09:09:40.816684 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:40.816609    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-04934e-548676699b-sctv8" podStartSLOduration=1.730779308 podStartE2EDuration="3.816590532s" totalImagesPullingTime="2.085811224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:09:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:09:40.813101918 +0000 UTC m=+1805.082165353" watchObservedRunningTime="2026-11-03 09:09:40.816590532 +0000 UTC m=+1805.085653971"
Nov 03 09:09:44.668920 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:44.668885    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:09:46.938158 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:09:46.938123    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:09:51.705240 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:51.705211    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-04934e-548676699b-sctv8"
Nov 03 09:09:52.054732 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:52.054649    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4417a34e-78fxn"]
Nov 03 09:09:52.059352 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:52.059330    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4417a34e-78fxn"]
Nov 03 09:09:53.052490 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:53.052454    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92118aae-2jb7c"]
Nov 03 09:09:53.102393 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:53.102363    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92118aae-2jb7c"]
Nov 03 09:09:53.871010 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:53.870976    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087" path="/var/lib/kubelet/pods/72b1bb1e-d9d4-4fe7-a2e8-dd0d012c2087/volumes"
Nov 03 09:09:53.871421 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:09:53.871407    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6" path="/var/lib/kubelet/pods/7b20c997-e1f9-4f94-9c0e-9d03e7d0ecb6/volumes"
Nov 03 09:10:03.508526 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:03.508491    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"]
Nov 03 09:10:03.509018 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:03.508724    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w" podUID="4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5" containerName="istio-proxy" containerID="cri-o://086e7f58b8c853aa1017057f6e20f4082bc9f3ef4d0f05d28356a516af58eb57" gracePeriod=30
Nov 03 09:10:08.781200 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:08.781138    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"
Nov 03 09:10:08.860347 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:08.860317    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-credential-socket\") pod \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\" (UID: \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\") "
Nov 03 09:10:08.860519 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:08.860403    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-workload-certs\") pod \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\" (UID: \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\") "
Nov 03 09:10:08.860519 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:08.860455    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-istio-token\" (UniqueName: \"kubernetes.io/projected/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-istio-token\") pod \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\" (UID: \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\") "
Nov 03 09:10:08.860519 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:08.860490    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-istio-podinfo\") pod \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\" (UID: \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\") "
Nov 03 09:10:08.860709 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:08.860520    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-istio-envoy\") pod \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\" (UID: \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\") "
Nov 03 09:10:08.860709 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:08.860554    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-credential-socket" pod "4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5" (UID: "4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:10:08.860709 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:08.860569    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-istio-data\") pod \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\" (UID: \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\") "
Nov 03 09:10:08.860709 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:08.860600    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-istiod-ca-cert\") pod \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\" (UID: \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\") "
Nov 03 09:10:08.860709 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:08.860701    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-workload-socket\") pod \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\" (UID: \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\") "
Nov 03 09:10:08.861007 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:08.860751    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-kube-api-access-wkdfj\" (UniqueName: \"kubernetes.io/projected/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-kube-api-access-wkdfj\") pod \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\" (UID: \"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5\") "
Nov 03 09:10:08.861007 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:08.860809    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-workload-certs" pod "4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5" (UID: "4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:10:08.861007 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:08.860852    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-istio-data" pod "4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5" (UID: "4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:10:08.861139 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:08.861011    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-workload-socket" pod "4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5" (UID: "4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:10:08.861139 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:08.861055    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-istiod-ca-cert" pod "4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5" (UID: "4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:10:08.861527 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:08.861506    4473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-workload-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:10:08.861584 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:08.861533    4473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-istio-data\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:10:08.861584 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:08.861547    4473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-istiod-ca-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:10:08.861584 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:08.861562    4473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-workload-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:10:08.861584 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:08.861577    4473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-credential-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:10:08.862903 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:08.862879    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-istio-podinfo" pod "4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5" (UID: "4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 09:10:08.863019 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:08.862936    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-istio-token" pod "4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5" (UID: "4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:10:08.863375 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:08.863357    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-istio-envoy" pod "4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5" (UID: "4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:10:08.863449 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:08.863427    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-kube-api-access-wkdfj" pod "4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5" (UID: "4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5"). InnerVolumeSpecName "kube-api-access-wkdfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:10:08.935597 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:08.935561    4473 generic.go:356] "Generic (PLEG): container finished" podID="4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5" containerID="086e7f58b8c853aa1017057f6e20f4082bc9f3ef4d0f05d28356a516af58eb57" exitCode=0
Nov 03 09:10:08.935735 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:08.935609    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"
Nov 03 09:10:08.935735 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:08.935651    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w" event={"ID":"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5","Type":"ContainerDied","Data":"086e7f58b8c853aa1017057f6e20f4082bc9f3ef4d0f05d28356a516af58eb57"}
Nov 03 09:10:08.935735 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:08.935717    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w" event={"ID":"4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5","Type":"ContainerDied","Data":"da768deed4b0e054e6c49e7b7d3810f4371c46306db7b316cd3cf48072d79a5b"}
Nov 03 09:10:08.935906 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:08.935747    4473 scope.go:122] "RemoveContainer" containerID="086e7f58b8c853aa1017057f6e20f4082bc9f3ef4d0f05d28356a516af58eb57"
Nov 03 09:10:08.946501 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:08.946485    4473 scope.go:122] "RemoveContainer" containerID="a3d29a7c4944cc360ce95640deabd3a70173b062a18f843efad97f5e894c3b01"
Nov 03 09:10:08.958479 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:08.958413    4473 scope.go:122] "RemoveContainer" containerID="086e7f58b8c853aa1017057f6e20f4082bc9f3ef4d0f05d28356a516af58eb57"
Nov 03 09:10:08.958740 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:10:08.958720    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"086e7f58b8c853aa1017057f6e20f4082bc9f3ef4d0f05d28356a516af58eb57\": container with ID starting with 086e7f58b8c853aa1017057f6e20f4082bc9f3ef4d0f05d28356a516af58eb57 not found: ID does not exist" containerID="086e7f58b8c853aa1017057f6e20f4082bc9f3ef4d0f05d28356a516af58eb57"
Nov 03 09:10:08.958809 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:08.958748    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"086e7f58b8c853aa1017057f6e20f4082bc9f3ef4d0f05d28356a516af58eb57"} err="failed to get container status \"086e7f58b8c853aa1017057f6e20f4082bc9f3ef4d0f05d28356a516af58eb57\": rpc error: code = NotFound desc = could not find container \"086e7f58b8c853aa1017057f6e20f4082bc9f3ef4d0f05d28356a516af58eb57\": container with ID starting with 086e7f58b8c853aa1017057f6e20f4082bc9f3ef4d0f05d28356a516af58eb57 not found: ID does not exist"
Nov 03 09:10:08.958809 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:08.958765    4473 scope.go:122] "RemoveContainer" containerID="a3d29a7c4944cc360ce95640deabd3a70173b062a18f843efad97f5e894c3b01"
Nov 03 09:10:08.959387 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:10:08.959364    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3d29a7c4944cc360ce95640deabd3a70173b062a18f843efad97f5e894c3b01\": container with ID starting with a3d29a7c4944cc360ce95640deabd3a70173b062a18f843efad97f5e894c3b01 not found: ID does not exist" containerID="a3d29a7c4944cc360ce95640deabd3a70173b062a18f843efad97f5e894c3b01"
Nov 03 09:10:08.959450 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:08.959398    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3d29a7c4944cc360ce95640deabd3a70173b062a18f843efad97f5e894c3b01"} err="failed to get container status \"a3d29a7c4944cc360ce95640deabd3a70173b062a18f843efad97f5e894c3b01\": rpc error: code = NotFound desc = could not find container \"a3d29a7c4944cc360ce95640deabd3a70173b062a18f843efad97f5e894c3b01\": container with ID starting with a3d29a7c4944cc360ce95640deabd3a70173b062a18f843efad97f5e894c3b01 not found: ID does not exist"
Nov 03 09:10:08.964295 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:08.964271    4473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-istio-token\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:10:08.964384 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:08.964301    4473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-istio-podinfo\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:10:08.964384 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:08.964316    4473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-istio-envoy\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:10:08.964384 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:08.964331    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkdfj\" (UniqueName: \"kubernetes.io/projected/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5-kube-api-access-wkdfj\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:10:09.066578 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:09.066515    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"]
Nov 03 09:10:09.073358 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:09.073335    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-eb9cd2-openshift-default-5cf46cbf89-j2k5w"]
Nov 03 09:10:09.871116 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:09.871086    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5" path="/var/lib/kubelet/pods/4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5/volumes"
Nov 03 09:10:16.937538 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:10:16.937462    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:10:21.499606 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:21.499578    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:10:22.888539 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:22.888506    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7378-846739"]
Nov 03 09:10:22.889171 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:22.889155    4473 cpu_manager.go:414] "containerMap: removing container" podUID="4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5" containerName="istio-proxy"
Nov 03 09:10:22.889218 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:22.889173    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5" containerName="istio-proxy"
Nov 03 09:10:22.889218 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:22.889195    4473 cpu_manager.go:414] "containerMap: removing container" podUID="4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5" containerName="istio-proxy"
Nov 03 09:10:22.889218 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:22.889201    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5" containerName="istio-proxy"
Nov 03 09:10:22.889335 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:22.889285    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5" containerName="istio-proxy"
Nov 03 09:10:22.889335 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:22.889296    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e4cf07f-a8e3-4cda-96d9-7e0b8f331bc5" containerName="istio-proxy"
Nov 03 09:10:22.893750 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:22.893736    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7378-846739"
Nov 03 09:10:22.937661 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:22.937630    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6tq6\" (UniqueName: \"kubernetes.io/projected/4e0a519d-f373-4cb8-8a66-9c844747d296-kube-api-access-s6tq6\") pod \"test-curl-7378-846739\" (UID: \"4e0a519d-f373-4cb8-8a66-9c844747d296\") " pod="opendatahub/test-curl-7378-846739"
Nov 03 09:10:22.947915 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:22.947892    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7378-846739"]
Nov 03 09:10:22.994695 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:22.994669    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-77bkh\""
Nov 03 09:10:23.038711 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:23.038672    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6tq6\" (UniqueName: \"kubernetes.io/projected/4e0a519d-f373-4cb8-8a66-9c844747d296-kube-api-access-s6tq6\") pod \"test-curl-7378-846739\" (UID: \"4e0a519d-f373-4cb8-8a66-9c844747d296\") " pod="opendatahub/test-curl-7378-846739"
Nov 03 09:10:23.099147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:23.099118    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6tq6\" (UniqueName: \"kubernetes.io/projected/4e0a519d-f373-4cb8-8a66-9c844747d296-kube-api-access-s6tq6\") pod \"test-curl-7378-846739\" (UID: \"4e0a519d-f373-4cb8-8a66-9c844747d296\") " pod="opendatahub/test-curl-7378-846739"
Nov 03 09:10:23.205489 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:23.205416    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7378-846739"
Nov 03 09:10:23.578180 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:23.578149    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7378-846739"]
Nov 03 09:10:23.578995 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:10:23.578967    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e0a519d_f373_4cb8_8a66_9c844747d296.slice/crio-d679248f8ca359c1541c2486e45766136f0d5d7e4b01ef86bbcb46cc69f1a670 WatchSource:0}: Error finding container d679248f8ca359c1541c2486e45766136f0d5d7e4b01ef86bbcb46cc69f1a670: Status 404 returned error can't find the container with id d679248f8ca359c1541c2486e45766136f0d5d7e4b01ef86bbcb46cc69f1a670
Nov 03 09:10:24.034019 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:24.033983    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7378-846739" event={"ID":"4e0a519d-f373-4cb8-8a66-9c844747d296","Type":"ContainerStarted","Data":"fa67d4fa9881ee7c5176ded2c17879470d406e599bf3973764252e378a849526"}
Nov 03 09:10:24.034019 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:24.034023    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7378-846739" event={"ID":"4e0a519d-f373-4cb8-8a66-9c844747d296","Type":"ContainerStarted","Data":"d679248f8ca359c1541c2486e45766136f0d5d7e4b01ef86bbcb46cc69f1a670"}
Nov 03 09:10:24.150766 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:24.150712    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7378-846739" podStartSLOduration=2.150698068 podStartE2EDuration="2.150698068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:10:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:10:24.14758862 +0000 UTC m=+1848.416652055" watchObservedRunningTime="2026-11-03 09:10:24.150698068 +0000 UTC m=+1848.419761502"
Nov 03 09:10:25.574570 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:25.574539    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7378-846739"]
Nov 03 09:10:25.578365 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:25.578336    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7378-846739"]
Nov 03 09:10:26.046965 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:26.046893    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7378-846739" podUID="4e0a519d-f373-4cb8-8a66-9c844747d296" containerName="test-curl-7378-846739" containerID="cri-o://fa67d4fa9881ee7c5176ded2c17879470d406e599bf3973764252e378a849526" gracePeriod=2
Nov 03 09:10:28.293050 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:28.293020    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7378-846739"
Nov 03 09:10:28.401989 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:28.401933    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e0a519d-f373-4cb8-8a66-9c844747d296-kube-api-access-s6tq6\" (UniqueName: \"kubernetes.io/projected/4e0a519d-f373-4cb8-8a66-9c844747d296-kube-api-access-s6tq6\") pod \"4e0a519d-f373-4cb8-8a66-9c844747d296\" (UID: \"4e0a519d-f373-4cb8-8a66-9c844747d296\") "
Nov 03 09:10:28.404052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:28.404029    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e0a519d-f373-4cb8-8a66-9c844747d296-kube-api-access-s6tq6" pod "4e0a519d-f373-4cb8-8a66-9c844747d296" (UID: "4e0a519d-f373-4cb8-8a66-9c844747d296"). InnerVolumeSpecName "kube-api-access-s6tq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:10:28.503849 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:28.503813    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6tq6\" (UniqueName: \"kubernetes.io/projected/4e0a519d-f373-4cb8-8a66-9c844747d296-kube-api-access-s6tq6\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:10:29.067393 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:29.067353    4473 generic.go:356] "Generic (PLEG): container finished" podID="4e0a519d-f373-4cb8-8a66-9c844747d296" containerID="fa67d4fa9881ee7c5176ded2c17879470d406e599bf3973764252e378a849526" exitCode=137
Nov 03 09:10:29.067393 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:29.067393    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7378-846739"
Nov 03 09:10:29.067647 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:29.067426    4473 scope.go:122] "RemoveContainer" containerID="fa67d4fa9881ee7c5176ded2c17879470d406e599bf3973764252e378a849526"
Nov 03 09:10:29.077011 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:29.076991    4473 scope.go:122] "RemoveContainer" containerID="fa67d4fa9881ee7c5176ded2c17879470d406e599bf3973764252e378a849526"
Nov 03 09:10:29.077266 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:10:29.077248    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa67d4fa9881ee7c5176ded2c17879470d406e599bf3973764252e378a849526\": container with ID starting with fa67d4fa9881ee7c5176ded2c17879470d406e599bf3973764252e378a849526 not found: ID does not exist" containerID="fa67d4fa9881ee7c5176ded2c17879470d406e599bf3973764252e378a849526"
Nov 03 09:10:29.077304 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:29.077276    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa67d4fa9881ee7c5176ded2c17879470d406e599bf3973764252e378a849526"} err="failed to get container status \"fa67d4fa9881ee7c5176ded2c17879470d406e599bf3973764252e378a849526\": rpc error: code = NotFound desc = could not find container \"fa67d4fa9881ee7c5176ded2c17879470d406e599bf3973764252e378a849526\": container with ID starting with fa67d4fa9881ee7c5176ded2c17879470d406e599bf3973764252e378a849526 not found: ID does not exist"
Nov 03 09:10:29.126965 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:10:29.126913    4473 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7378-846739\" is forbidden: User \"system:node:ip-10-0-139-73.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-73.us-west-1.compute.internal' and this object" podUID="4e0a519d-f373-4cb8-8a66-9c844747d296" pod="opendatahub/test-curl-7378-846739"
Nov 03 09:10:29.870246 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:29.870213    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e0a519d-f373-4cb8-8a66-9c844747d296" path="/var/lib/kubelet/pods/4e0a519d-f373-4cb8-8a66-9c844747d296/volumes"
Nov 03 09:10:31.037046 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:31.037012    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-b0c653f4/e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss"]
Nov 03 09:10:34.054631 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:34.054587    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1993359e-xfx7p"]
Nov 03 09:10:34.106334 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:34.106303    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1993359e-xfx7p"]
Nov 03 09:10:35.871301 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:35.871271    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9347af0f-ab7f-4949-8642-a92f82388f44" path="/var/lib/kubelet/pods/9347af0f-ab7f-4949-8642-a92f82388f44/volumes"
Nov 03 09:10:36.610755 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:36.606679    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-8566fbf99f-47ltl"]
Nov 03 09:10:36.748372 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:36.748339    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"]
Nov 03 09:10:36.749054 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:36.749031    4473 cpu_manager.go:414] "containerMap: removing container" podUID="4e0a519d-f373-4cb8-8a66-9c844747d296" containerName="test-curl-7378-846739"
Nov 03 09:10:36.749054 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:36.749054    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e0a519d-f373-4cb8-8a66-9c844747d296" containerName="test-curl-7378-846739"
Nov 03 09:10:36.749262 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:36.749213    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e0a519d-f373-4cb8-8a66-9c844747d296" containerName="test-curl-7378-846739"
Nov 03 09:10:36.753895 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:36.753877    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"
Nov 03 09:10:36.813967 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:36.813914    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"]
Nov 03 09:10:36.817147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:36.817115    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/990b9e6c-86b3-4bed-beee-4a1537a8c379-istio-data\") pod \"e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5\" (UID: \"990b9e6c-86b3-4bed-beee-4a1537a8c379\") " pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"
Nov 03 09:10:36.817270 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:36.817237    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/990b9e6c-86b3-4bed-beee-4a1537a8c379-istiod-ca-cert\") pod \"e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5\" (UID: \"990b9e6c-86b3-4bed-beee-4a1537a8c379\") " pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"
Nov 03 09:10:36.817353 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:36.817318    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/990b9e6c-86b3-4bed-beee-4a1537a8c379-istio-token\") pod \"e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5\" (UID: \"990b9e6c-86b3-4bed-beee-4a1537a8c379\") " pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"
Nov 03 09:10:36.817416 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:36.817355    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/990b9e6c-86b3-4bed-beee-4a1537a8c379-istio-envoy\") pod \"e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5\" (UID: \"990b9e6c-86b3-4bed-beee-4a1537a8c379\") " pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"
Nov 03 09:10:36.817416 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:36.817375    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/990b9e6c-86b3-4bed-beee-4a1537a8c379-workload-socket\") pod \"e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5\" (UID: \"990b9e6c-86b3-4bed-beee-4a1537a8c379\") " pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"
Nov 03 09:10:36.817416 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:36.817395    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/990b9e6c-86b3-4bed-beee-4a1537a8c379-workload-certs\") pod \"e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5\" (UID: \"990b9e6c-86b3-4bed-beee-4a1537a8c379\") " pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"
Nov 03 09:10:36.817556 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:36.817441    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/990b9e6c-86b3-4bed-beee-4a1537a8c379-istio-podinfo\") pod \"e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5\" (UID: \"990b9e6c-86b3-4bed-beee-4a1537a8c379\") " pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"
Nov 03 09:10:36.817556 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:36.817469    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htftc\" (UniqueName: \"kubernetes.io/projected/990b9e6c-86b3-4bed-beee-4a1537a8c379-kube-api-access-htftc\") pod \"e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5\" (UID: \"990b9e6c-86b3-4bed-beee-4a1537a8c379\") " pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"
Nov 03 09:10:36.817556 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:36.817508    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/990b9e6c-86b3-4bed-beee-4a1537a8c379-credential-socket\") pod \"e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5\" (UID: \"990b9e6c-86b3-4bed-beee-4a1537a8c379\") " pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"
Nov 03 09:10:36.848498 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:36.848466    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-c78f2b-openshift-default-dockercfg-2b7l6\""
Nov 03 09:10:36.942873 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:36.942831    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/990b9e6c-86b3-4bed-beee-4a1537a8c379-credential-socket\") pod \"e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5\" (UID: \"990b9e6c-86b3-4bed-beee-4a1537a8c379\") " pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"
Nov 03 09:10:36.944808 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:36.944780    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/990b9e6c-86b3-4bed-beee-4a1537a8c379-istio-data\") pod \"e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5\" (UID: \"990b9e6c-86b3-4bed-beee-4a1537a8c379\") " pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"
Nov 03 09:10:36.956195 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:36.956162    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/990b9e6c-86b3-4bed-beee-4a1537a8c379-istiod-ca-cert\") pod \"e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5\" (UID: \"990b9e6c-86b3-4bed-beee-4a1537a8c379\") " pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"
Nov 03 09:10:36.971500 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:36.971473    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/990b9e6c-86b3-4bed-beee-4a1537a8c379-istio-token\") pod \"e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5\" (UID: \"990b9e6c-86b3-4bed-beee-4a1537a8c379\") " pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"
Nov 03 09:10:36.971752 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:36.971737    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/990b9e6c-86b3-4bed-beee-4a1537a8c379-istio-envoy\") pod \"e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5\" (UID: \"990b9e6c-86b3-4bed-beee-4a1537a8c379\") " pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"
Nov 03 09:10:36.971874 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:36.971861    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/990b9e6c-86b3-4bed-beee-4a1537a8c379-workload-socket\") pod \"e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5\" (UID: \"990b9e6c-86b3-4bed-beee-4a1537a8c379\") " pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"
Nov 03 09:10:36.971999 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:36.971986    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/990b9e6c-86b3-4bed-beee-4a1537a8c379-workload-certs\") pod \"e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5\" (UID: \"990b9e6c-86b3-4bed-beee-4a1537a8c379\") " pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"
Nov 03 09:10:36.972216 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:36.972202    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/990b9e6c-86b3-4bed-beee-4a1537a8c379-istio-podinfo\") pod \"e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5\" (UID: \"990b9e6c-86b3-4bed-beee-4a1537a8c379\") " pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"
Nov 03 09:10:36.973043 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:36.973018    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htftc\" (UniqueName: \"kubernetes.io/projected/990b9e6c-86b3-4bed-beee-4a1537a8c379-kube-api-access-htftc\") pod \"e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5\" (UID: \"990b9e6c-86b3-4bed-beee-4a1537a8c379\") " pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"
Nov 03 09:10:36.973294 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:36.945295    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/990b9e6c-86b3-4bed-beee-4a1537a8c379-istio-data\") pod \"e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5\" (UID: \"990b9e6c-86b3-4bed-beee-4a1537a8c379\") " pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"
Nov 03 09:10:36.973391 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:36.943920    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/990b9e6c-86b3-4bed-beee-4a1537a8c379-credential-socket\") pod \"e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5\" (UID: \"990b9e6c-86b3-4bed-beee-4a1537a8c379\") " pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"
Nov 03 09:10:36.977307 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:36.973727    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/990b9e6c-86b3-4bed-beee-4a1537a8c379-workload-socket\") pod \"e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5\" (UID: \"990b9e6c-86b3-4bed-beee-4a1537a8c379\") " pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"
Nov 03 09:10:36.977460 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:36.974516    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/990b9e6c-86b3-4bed-beee-4a1537a8c379-workload-certs\") pod \"e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5\" (UID: \"990b9e6c-86b3-4bed-beee-4a1537a8c379\") " pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"
Nov 03 09:10:36.984215 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:36.971325    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/990b9e6c-86b3-4bed-beee-4a1537a8c379-istiod-ca-cert\") pod \"e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5\" (UID: \"990b9e6c-86b3-4bed-beee-4a1537a8c379\") " pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"
Nov 03 09:10:36.988057 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:36.988028    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/990b9e6c-86b3-4bed-beee-4a1537a8c379-istio-envoy\") pod \"e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5\" (UID: \"990b9e6c-86b3-4bed-beee-4a1537a8c379\") " pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"
Nov 03 09:10:37.009969 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:37.006128    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/990b9e6c-86b3-4bed-beee-4a1537a8c379-istio-podinfo\") pod \"e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5\" (UID: \"990b9e6c-86b3-4bed-beee-4a1537a8c379\") " pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"
Nov 03 09:10:37.036509 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:37.036478    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htftc\" (UniqueName: \"kubernetes.io/projected/990b9e6c-86b3-4bed-beee-4a1537a8c379-kube-api-access-htftc\") pod \"e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5\" (UID: \"990b9e6c-86b3-4bed-beee-4a1537a8c379\") " pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"
Nov 03 09:10:37.036848 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:37.036828    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/990b9e6c-86b3-4bed-beee-4a1537a8c379-istio-token\") pod \"e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5\" (UID: \"990b9e6c-86b3-4bed-beee-4a1537a8c379\") " pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"
Nov 03 09:10:37.075803 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:37.075752    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"
Nov 03 09:10:37.237967 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:37.237849    4473 scope.go:122] "RemoveContainer" containerID="167c3b8bf7f67551ef77a0d519a8a47a9a7c09a2a554625aebd7f695fb0927e6"
Nov 03 09:10:37.549426 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:37.549402    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"]
Nov 03 09:10:37.550440 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:10:37.550415    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod990b9e6c_86b3_4bed_beee_4a1537a8c379.slice/crio-706fbe0c476c6737d47087c033ec46d2c2947fcf19e931672be870545e829cd2 WatchSource:0}: Error finding container 706fbe0c476c6737d47087c033ec46d2c2947fcf19e931672be870545e829cd2: Status 404 returned error can't find the container with id 706fbe0c476c6737d47087c033ec46d2c2947fcf19e931672be870545e829cd2
Nov 03 09:10:37.553008 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:37.552980    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:10:37.553069 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:37.553039    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:10:37.553108 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:37.553075    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:10:38.135403 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:38.135366    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5" event={"ID":"990b9e6c-86b3-4bed-beee-4a1537a8c379","Type":"ContainerStarted","Data":"f96d73b89cffe97819b840bd987c4c8756144953435cd01964c79af15f49dc23"}
Nov 03 09:10:38.136020 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:38.135996    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5" event={"ID":"990b9e6c-86b3-4bed-beee-4a1537a8c379","Type":"ContainerStarted","Data":"706fbe0c476c6737d47087c033ec46d2c2947fcf19e931672be870545e829cd2"}
Nov 03 09:10:39.076085 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:39.076056    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"
Nov 03 09:10:39.081679 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:39.081656    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"
Nov 03 09:10:39.141617 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:39.141584    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"
Nov 03 09:10:39.142457 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:39.142434    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"
Nov 03 09:10:39.250698 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:39.250647    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5" podStartSLOduration=3.250635244 podStartE2EDuration="3.250635244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:10:38.270291636 +0000 UTC m=+1862.539355071" watchObservedRunningTime="2026-11-03 09:10:39.250635244 +0000 UTC m=+1863.519698677"
Nov 03 09:10:41.611305 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:41.611270    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-8566fbf99f-47ltl" podUID="ba3d1e4b-5057-429c-9c89-0045b964f1df" containerName="payload-pre-processing" containerID="cri-o://1503917794ce0d39cb136574badc6828973fa0dbbf2d8e8054383bd1cb93b564" gracePeriod=25
Nov 03 09:10:41.880265 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:41.880240    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-8566fbf99f-47ltl"
Nov 03 09:10:41.958504 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:41.958465    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba3d1e4b-5057-429c-9c89-0045b964f1df-kube-api-access-5h27d\" (UniqueName: \"kubernetes.io/projected/ba3d1e4b-5057-429c-9c89-0045b964f1df-kube-api-access-5h27d\") pod \"ba3d1e4b-5057-429c-9c89-0045b964f1df\" (UID: \"ba3d1e4b-5057-429c-9c89-0045b964f1df\") "
Nov 03 09:10:41.958729 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:41.958710    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba3d1e4b-5057-429c-9c89-0045b964f1df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba3d1e4b-5057-429c-9c89-0045b964f1df-plugins-config-volume\") pod \"ba3d1e4b-5057-429c-9c89-0045b964f1df\" (UID: \"ba3d1e4b-5057-429c-9c89-0045b964f1df\") "
Nov 03 09:10:41.959158 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:41.959133    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba3d1e4b-5057-429c-9c89-0045b964f1df-plugins-config-volume" pod "ba3d1e4b-5057-429c-9c89-0045b964f1df" (UID: "ba3d1e4b-5057-429c-9c89-0045b964f1df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:10:41.959275 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:41.959259    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba3d1e4b-5057-429c-9c89-0045b964f1df-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:10:41.960990 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:41.960966    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba3d1e4b-5057-429c-9c89-0045b964f1df-kube-api-access-5h27d" pod "ba3d1e4b-5057-429c-9c89-0045b964f1df" (UID: "ba3d1e4b-5057-429c-9c89-0045b964f1df"). InnerVolumeSpecName "kube-api-access-5h27d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:10:42.060356 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:42.060326    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h27d\" (UniqueName: \"kubernetes.io/projected/ba3d1e4b-5057-429c-9c89-0045b964f1df-kube-api-access-5h27d\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:10:42.160997 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:42.160963    4473 generic.go:356] "Generic (PLEG): container finished" podID="ba3d1e4b-5057-429c-9c89-0045b964f1df" containerID="1503917794ce0d39cb136574badc6828973fa0dbbf2d8e8054383bd1cb93b564" exitCode=0
Nov 03 09:10:42.161178 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:42.161034    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-8566fbf99f-47ltl"
Nov 03 09:10:42.161178 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:42.161063    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-8566fbf99f-47ltl" event={"ID":"ba3d1e4b-5057-429c-9c89-0045b964f1df","Type":"ContainerDied","Data":"1503917794ce0d39cb136574badc6828973fa0dbbf2d8e8054383bd1cb93b564"}
Nov 03 09:10:42.161178 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:42.161117    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-8566fbf99f-47ltl" event={"ID":"ba3d1e4b-5057-429c-9c89-0045b964f1df","Type":"ContainerDied","Data":"7de5a670f4460ff05870ac445ce89f81038be4387342367372f603ffdfdcdca5"}
Nov 03 09:10:42.161178 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:42.161138    4473 scope.go:122] "RemoveContainer" containerID="1503917794ce0d39cb136574badc6828973fa0dbbf2d8e8054383bd1cb93b564"
Nov 03 09:10:42.172411 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:42.172391    4473 scope.go:122] "RemoveContainer" containerID="1503917794ce0d39cb136574badc6828973fa0dbbf2d8e8054383bd1cb93b564"
Nov 03 09:10:42.172708 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:10:42.172687    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1503917794ce0d39cb136574badc6828973fa0dbbf2d8e8054383bd1cb93b564\": container with ID starting with 1503917794ce0d39cb136574badc6828973fa0dbbf2d8e8054383bd1cb93b564 not found: ID does not exist" containerID="1503917794ce0d39cb136574badc6828973fa0dbbf2d8e8054383bd1cb93b564"
Nov 03 09:10:42.172809 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:42.172713    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1503917794ce0d39cb136574badc6828973fa0dbbf2d8e8054383bd1cb93b564"} err="failed to get container status \"1503917794ce0d39cb136574badc6828973fa0dbbf2d8e8054383bd1cb93b564\": rpc error: code = NotFound desc = could not find container \"1503917794ce0d39cb136574badc6828973fa0dbbf2d8e8054383bd1cb93b564\": container with ID starting with 1503917794ce0d39cb136574badc6828973fa0dbbf2d8e8054383bd1cb93b564 not found: ID does not exist"
Nov 03 09:10:42.346425 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:42.346390    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-8566fbf99f-47ltl"]
Nov 03 09:10:42.351897 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:42.351873    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4ae3db-8566fbf99f-47ltl"]
Nov 03 09:10:43.870739 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:43.870696    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba3d1e4b-5057-429c-9c89-0045b964f1df" path="/var/lib/kubelet/pods/ba3d1e4b-5057-429c-9c89-0045b964f1df/volumes"
Nov 03 09:10:46.063975 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:46.063148    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-b0c653f4/e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss" podUID="7213a5c7-1610-4205-8d73-3d096c44c277" containerName="main" containerID="cri-o://0c62da9e976f8789b9fcfb57a65387671cb3ef5ee47998ee6fd0fa7d814dd8fe" gracePeriod=45
Nov 03 09:10:46.201846 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:46.201810    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-b0c653f4/e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss" podUID="7213a5c7-1610-4205-8d73-3d096c44c277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.96:8000/ready\": dial tcp 10.133.0.96:8000: connect: connection refused"
Nov 03 09:10:46.209901 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:46.209871    4473 generic.go:356] "Generic (PLEG): container finished" podID="7213a5c7-1610-4205-8d73-3d096c44c277" containerID="0c62da9e976f8789b9fcfb57a65387671cb3ef5ee47998ee6fd0fa7d814dd8fe" exitCode=0
Nov 03 09:10:46.210033 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:46.209920    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b0c653f4/e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss" event={"ID":"7213a5c7-1610-4205-8d73-3d096c44c277","Type":"ContainerDied","Data":"0c62da9e976f8789b9fcfb57a65387671cb3ef5ee47998ee6fd0fa7d814dd8fe"}
Nov 03 09:10:46.351194 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:46.351171    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b0c653f4/e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss"
Nov 03 09:10:46.440652 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:46.440626    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7213a5c7-1610-4205-8d73-3d096c44c277-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7213a5c7-1610-4205-8d73-3d096c44c277-dshm\") pod \"7213a5c7-1610-4205-8d73-3d096c44c277\" (UID: \"7213a5c7-1610-4205-8d73-3d096c44c277\") "
Nov 03 09:10:46.440814 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:46.440674    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7213a5c7-1610-4205-8d73-3d096c44c277-home\" (UniqueName: \"kubernetes.io/empty-dir/7213a5c7-1610-4205-8d73-3d096c44c277-home\") pod \"7213a5c7-1610-4205-8d73-3d096c44c277\" (UID: \"7213a5c7-1610-4205-8d73-3d096c44c277\") "
Nov 03 09:10:46.440814 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:46.440712    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7213a5c7-1610-4205-8d73-3d096c44c277-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7213a5c7-1610-4205-8d73-3d096c44c277-tmp-dir\") pod \"7213a5c7-1610-4205-8d73-3d096c44c277\" (UID: \"7213a5c7-1610-4205-8d73-3d096c44c277\") "
Nov 03 09:10:46.440814 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:46.440749    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7213a5c7-1610-4205-8d73-3d096c44c277-kube-api-access-j8gql\" (UniqueName: \"kubernetes.io/projected/7213a5c7-1610-4205-8d73-3d096c44c277-kube-api-access-j8gql\") pod \"7213a5c7-1610-4205-8d73-3d096c44c277\" (UID: \"7213a5c7-1610-4205-8d73-3d096c44c277\") "
Nov 03 09:10:46.440814 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:46.440766    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7213a5c7-1610-4205-8d73-3d096c44c277-tls-certs\" (UniqueName: \"kubernetes.io/secret/7213a5c7-1610-4205-8d73-3d096c44c277-tls-certs\") pod \"7213a5c7-1610-4205-8d73-3d096c44c277\" (UID: \"7213a5c7-1610-4205-8d73-3d096c44c277\") "
Nov 03 09:10:46.441198 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:46.440820    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7213a5c7-1610-4205-8d73-3d096c44c277-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7213a5c7-1610-4205-8d73-3d096c44c277-model-cache\") pod \"7213a5c7-1610-4205-8d73-3d096c44c277\" (UID: \"7213a5c7-1610-4205-8d73-3d096c44c277\") "
Nov 03 09:10:46.441198 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:46.440938    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7213a5c7-1610-4205-8d73-3d096c44c277-home" pod "7213a5c7-1610-4205-8d73-3d096c44c277" (UID: "7213a5c7-1610-4205-8d73-3d096c44c277"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:10:46.441198 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:46.441098    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7213a5c7-1610-4205-8d73-3d096c44c277-tmp-dir" pod "7213a5c7-1610-4205-8d73-3d096c44c277" (UID: "7213a5c7-1610-4205-8d73-3d096c44c277"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:10:46.441453 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:46.441424    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7213a5c7-1610-4205-8d73-3d096c44c277-model-cache" pod "7213a5c7-1610-4205-8d73-3d096c44c277" (UID: "7213a5c7-1610-4205-8d73-3d096c44c277"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:10:46.441736 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:46.441452    4473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7213a5c7-1610-4205-8d73-3d096c44c277-home\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:10:46.441736 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:46.441637    4473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7213a5c7-1610-4205-8d73-3d096c44c277-tmp-dir\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:10:46.443290 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:46.443267    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7213a5c7-1610-4205-8d73-3d096c44c277-dshm" pod "7213a5c7-1610-4205-8d73-3d096c44c277" (UID: "7213a5c7-1610-4205-8d73-3d096c44c277"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:10:46.443290 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:46.443283    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7213a5c7-1610-4205-8d73-3d096c44c277-tls-certs" pod "7213a5c7-1610-4205-8d73-3d096c44c277" (UID: "7213a5c7-1610-4205-8d73-3d096c44c277"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:10:46.443410 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:46.443306    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7213a5c7-1610-4205-8d73-3d096c44c277-kube-api-access-j8gql" pod "7213a5c7-1610-4205-8d73-3d096c44c277" (UID: "7213a5c7-1610-4205-8d73-3d096c44c277"). InnerVolumeSpecName "kube-api-access-j8gql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:10:46.543234 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:46.543204    4473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7213a5c7-1610-4205-8d73-3d096c44c277-model-cache\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:10:46.543234 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:46.543232    4473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7213a5c7-1610-4205-8d73-3d096c44c277-dshm\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:10:46.543472 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:46.543246    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8gql\" (UniqueName: \"kubernetes.io/projected/7213a5c7-1610-4205-8d73-3d096c44c277-kube-api-access-j8gql\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:10:46.543472 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:46.543260    4473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7213a5c7-1610-4205-8d73-3d096c44c277-tls-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:10:46.938045 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:10:46.938007    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:10:47.217893 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:47.217817    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b0c653f4/e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss"
Nov 03 09:10:47.217893 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:47.217852    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b0c653f4/e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss" event={"ID":"7213a5c7-1610-4205-8d73-3d096c44c277","Type":"ContainerDied","Data":"b2e283007bcb5e73e5ff98af4daab344338818d4cf6b0e7077386fb1444fd8dd"}
Nov 03 09:10:47.218397 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:47.217904    4473 scope.go:122] "RemoveContainer" containerID="0c62da9e976f8789b9fcfb57a65387671cb3ef5ee47998ee6fd0fa7d814dd8fe"
Nov 03 09:10:47.388560 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:47.388530    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-b0c653f4/e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss"]
Nov 03 09:10:47.438787 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:47.438762    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-b0c653f4/e2e-lifecycle-model-b0c653f4-kserve-85ddfc5685-r7zss"]
Nov 03 09:10:47.916593 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:10:47.916553    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7213a5c7-1610-4205-8d73-3d096c44c277" path="/var/lib/kubelet/pods/7213a5c7-1610-4205-8d73-3d096c44c277/volumes"
Nov 03 09:11:08.726080 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:11:08.726050    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-840559-798b95d567-46skj"]
Nov 03 09:11:13.726894 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:11:13.726853    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-840559-798b95d567-46skj" podUID="048badaf-8a93-4e51-8308-2a75a72487f3" containerName="payload-pre-processing" containerID="cri-o://9ed25b860de7619100afdd4583fd9179c90749b3d7c802eff036b45c4a13e2ef" gracePeriod=25
Nov 03 09:11:14.095450 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:11:14.095425    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-840559-798b95d567-46skj"
Nov 03 09:11:14.191976 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:11:14.191899    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/048badaf-8a93-4e51-8308-2a75a72487f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048badaf-8a93-4e51-8308-2a75a72487f3-plugins-config-volume\") pod \"048badaf-8a93-4e51-8308-2a75a72487f3\" (UID: \"048badaf-8a93-4e51-8308-2a75a72487f3\") "
Nov 03 09:11:14.193366 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:11:14.192690    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/048badaf-8a93-4e51-8308-2a75a72487f3-plugins-config-volume" pod "048badaf-8a93-4e51-8308-2a75a72487f3" (UID: "048badaf-8a93-4e51-8308-2a75a72487f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:11:14.193553 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:11:14.193516    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/048badaf-8a93-4e51-8308-2a75a72487f3-kube-api-access-8ph74\" (UniqueName: \"kubernetes.io/projected/048badaf-8a93-4e51-8308-2a75a72487f3-kube-api-access-8ph74\") pod \"048badaf-8a93-4e51-8308-2a75a72487f3\" (UID: \"048badaf-8a93-4e51-8308-2a75a72487f3\") "
Nov 03 09:11:14.194264 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:11:14.194243    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048badaf-8a93-4e51-8308-2a75a72487f3-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:11:14.197462 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:11:14.197434    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/048badaf-8a93-4e51-8308-2a75a72487f3-kube-api-access-8ph74" pod "048badaf-8a93-4e51-8308-2a75a72487f3" (UID: "048badaf-8a93-4e51-8308-2a75a72487f3"). InnerVolumeSpecName "kube-api-access-8ph74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:11:14.301826 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:11:14.296667    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ph74\" (UniqueName: \"kubernetes.io/projected/048badaf-8a93-4e51-8308-2a75a72487f3-kube-api-access-8ph74\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:11:14.402531 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:11:14.402495    4473 generic.go:356] "Generic (PLEG): container finished" podID="048badaf-8a93-4e51-8308-2a75a72487f3" containerID="9ed25b860de7619100afdd4583fd9179c90749b3d7c802eff036b45c4a13e2ef" exitCode=0
Nov 03 09:11:14.402701 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:11:14.402551    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-840559-798b95d567-46skj" event={"ID":"048badaf-8a93-4e51-8308-2a75a72487f3","Type":"ContainerDied","Data":"9ed25b860de7619100afdd4583fd9179c90749b3d7c802eff036b45c4a13e2ef"}
Nov 03 09:11:14.402701 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:11:14.402553    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-840559-798b95d567-46skj"
Nov 03 09:11:14.402701 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:11:14.402617    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-840559-798b95d567-46skj" event={"ID":"048badaf-8a93-4e51-8308-2a75a72487f3","Type":"ContainerDied","Data":"61d9479b83960710e7f72c6249669676fc243dfca0e4a8e5af175698dbb72e4b"}
Nov 03 09:11:14.402701 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:11:14.402638    4473 scope.go:122] "RemoveContainer" containerID="9ed25b860de7619100afdd4583fd9179c90749b3d7c802eff036b45c4a13e2ef"
Nov 03 09:11:14.412804 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:11:14.412777    4473 scope.go:122] "RemoveContainer" containerID="9ed25b860de7619100afdd4583fd9179c90749b3d7c802eff036b45c4a13e2ef"
Nov 03 09:11:14.413137 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:11:14.413114    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ed25b860de7619100afdd4583fd9179c90749b3d7c802eff036b45c4a13e2ef\": container with ID starting with 9ed25b860de7619100afdd4583fd9179c90749b3d7c802eff036b45c4a13e2ef not found: ID does not exist" containerID="9ed25b860de7619100afdd4583fd9179c90749b3d7c802eff036b45c4a13e2ef"
Nov 03 09:11:14.413265 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:11:14.413144    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ed25b860de7619100afdd4583fd9179c90749b3d7c802eff036b45c4a13e2ef"} err="failed to get container status \"9ed25b860de7619100afdd4583fd9179c90749b3d7c802eff036b45c4a13e2ef\": rpc error: code = NotFound desc = could not find container \"9ed25b860de7619100afdd4583fd9179c90749b3d7c802eff036b45c4a13e2ef\": container with ID starting with 9ed25b860de7619100afdd4583fd9179c90749b3d7c802eff036b45c4a13e2ef not found: ID does not exist"
Nov 03 09:11:14.571890 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:11:14.571851    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-840559-798b95d567-46skj"]
Nov 03 09:11:14.577086 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:11:14.577056    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-840559-798b95d567-46skj"]
Nov 03 09:11:15.872103 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:11:15.872073    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="048badaf-8a93-4e51-8308-2a75a72487f3" path="/var/lib/kubelet/pods/048badaf-8a93-4e51-8308-2a75a72487f3/volumes"
Nov 03 09:11:16.939066 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:11:16.939034    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:11:25.023388 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:11:25.023351    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:11:37.289753 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:11:37.289721    4473 scope.go:122] "RemoveContainer" containerID="bd84facbf317ca9000e003ba96094b909fa19648493a4d962a2c5a83704f0c69"
Nov 03 09:11:37.299102 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:11:37.299081    4473 scope.go:122] "RemoveContainer" containerID="23946cddeec9a39b9dbf60111b26051b897ff6db73202e17a7486ba74db82061"
Nov 03 09:11:37.307663 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:11:37.307637    4473 scope.go:122] "RemoveContainer" containerID="f7faea6109fb78f5f36adccfeae4b736403a96444f93f888ebfc36a282b2c7c3"
Nov 03 09:11:46.937759 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:11:46.937669    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:12:11.687250 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:12:11.687216    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:12:16.963368 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:12:16.963333    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:12:22.052179 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:12:22.052151    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da7f4c25-cwtx8"]
Nov 03 09:12:22.069132 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:12:22.069101    4473 generic.go:356] "Generic (PLEG): container finished" podID="990b9e6c-86b3-4bed-beee-4a1537a8c379" containerID="f96d73b89cffe97819b840bd987c4c8756144953435cd01964c79af15f49dc23" exitCode=0
Nov 03 09:12:22.069256 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:12:22.069174    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5" event={"ID":"990b9e6c-86b3-4bed-beee-4a1537a8c379","Type":"ContainerDied","Data":"f96d73b89cffe97819b840bd987c4c8756144953435cd01964c79af15f49dc23"}
Nov 03 09:12:22.069614 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:12:22.069598    4473 scope.go:122] "RemoveContainer" containerID="f96d73b89cffe97819b840bd987c4c8756144953435cd01964c79af15f49dc23"
Nov 03 09:12:22.071126 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:12:22.071099    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:12:22.071198 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:12:22.071166    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:12:22.071233 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:12:22.071199    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:12:22.103855 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:12:22.103831    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da7f4c25-cwtx8"]
Nov 03 09:12:23.077453 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:12:23.077416    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5" event={"ID":"990b9e6c-86b3-4bed-beee-4a1537a8c379","Type":"ContainerStarted","Data":"2227037a2f8d66d69790cfe60615c22a2f8ff9f548043bd12848f413697a9968"}
Nov 03 09:12:23.873283 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:12:23.873253    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9b64db0-8da5-4bb5-a1dc-d4962912ad9f" path="/var/lib/kubelet/pods/d9b64db0-8da5-4bb5-a1dc-d4962912ad9f/volumes"
Nov 03 09:12:24.076711 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:12:24.076670    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"
Nov 03 09:12:24.077979 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:12:24.077957    4473 patch_prober.go:29] interesting pod/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 09:12:24.078328 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:12:24.078004    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5" podUID="990b9e6c-86b3-4bed-beee-4a1537a8c379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 09:12:24.141671 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:12:24.141582    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"
Nov 03 09:12:25.076746 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:12:25.076714    4473 patch_prober.go:29] interesting pod/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 09:12:25.076937 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:12:25.076764    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5" podUID="990b9e6c-86b3-4bed-beee-4a1537a8c379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 09:12:26.076086 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:12:26.076051    4473 patch_prober.go:29] interesting pod/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 09:12:26.076505 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:12:26.076101    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5" podUID="990b9e6c-86b3-4bed-beee-4a1537a8c379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 09:12:27.076815 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:12:27.076778    4473 patch_prober.go:29] interesting pod/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 09:12:27.077333 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:12:27.076835    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5" podUID="990b9e6c-86b3-4bed-beee-4a1537a8c379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 09:12:28.076245 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:12:28.076212    4473 patch_prober.go:29] interesting pod/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 03 09:12:28.076409 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:12:28.076265    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5" podUID="990b9e6c-86b3-4bed-beee-4a1537a8c379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 03 09:12:29.080882 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:12:29.080854    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"
Nov 03 09:12:29.081886 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:12:29.081870    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"
Nov 03 09:12:37.348474 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:12:37.348438    4473 scope.go:122] "RemoveContainer" containerID="86a3a6858ed6adc436573c725e9dcf611d641d21443d2cda69cd7663aa44cde5"
Nov 03 09:12:46.936503 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:12:46.936467    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:12:50.045905 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:12:50.045871    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:13:16.937175 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:13:16.937088    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:13:37.331149 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:37.326926    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"]
Nov 03 09:13:37.332117 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:37.332079    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4" podUID="1febda0d-e4ac-436c-863e-902751af88ca" containerName="istio-proxy" containerID="cri-o://171c40d5e73be093a52718481f53fce86703c829f1475be01709f718c6cef14d" gracePeriod=30
Nov 03 09:13:37.408025 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:37.398997    4473 scope.go:122] "RemoveContainer" containerID="aaebe709a5f6fc5e52fcfcdd99680f2ceb9b6c59c8b208d18e1764cfcc1ca351"
Nov 03 09:13:37.617799 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:37.617772    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:13:40.946414 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:40.946367    4473 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 09:13:40.946798 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:40.946446    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4" podUID="1febda0d-e4ac-436c-863e-902751af88ca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 09:13:42.587629 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:42.587597    4473 generic.go:356] "Generic (PLEG): container finished" podID="1febda0d-e4ac-436c-863e-902751af88ca" containerID="171c40d5e73be093a52718481f53fce86703c829f1475be01709f718c6cef14d" exitCode=0
Nov 03 09:13:42.588095 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:42.587644    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4" event={"ID":"1febda0d-e4ac-436c-863e-902751af88ca","Type":"ContainerDied","Data":"171c40d5e73be093a52718481f53fce86703c829f1475be01709f718c6cef14d"}
Nov 03 09:13:42.631667 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:42.631646    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"
Nov 03 09:13:42.791845 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:42.791769    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1febda0d-e4ac-436c-863e-902751af88ca-kube-api-access-9vqbx\" (UniqueName: \"kubernetes.io/projected/1febda0d-e4ac-436c-863e-902751af88ca-kube-api-access-9vqbx\") pod \"1febda0d-e4ac-436c-863e-902751af88ca\" (UID: \"1febda0d-e4ac-436c-863e-902751af88ca\") "
Nov 03 09:13:42.791845 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:42.791824    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1febda0d-e4ac-436c-863e-902751af88ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1febda0d-e4ac-436c-863e-902751af88ca-istio-envoy\") pod \"1febda0d-e4ac-436c-863e-902751af88ca\" (UID: \"1febda0d-e4ac-436c-863e-902751af88ca\") "
Nov 03 09:13:42.792100 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:42.791852    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1febda0d-e4ac-436c-863e-902751af88ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1febda0d-e4ac-436c-863e-902751af88ca-workload-certs\") pod \"1febda0d-e4ac-436c-863e-902751af88ca\" (UID: \"1febda0d-e4ac-436c-863e-902751af88ca\") "
Nov 03 09:13:42.792100 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:42.791889    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1febda0d-e4ac-436c-863e-902751af88ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1febda0d-e4ac-436c-863e-902751af88ca-istiod-ca-cert\") pod \"1febda0d-e4ac-436c-863e-902751af88ca\" (UID: \"1febda0d-e4ac-436c-863e-902751af88ca\") "
Nov 03 09:13:42.792100 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:42.791927    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1febda0d-e4ac-436c-863e-902751af88ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1febda0d-e4ac-436c-863e-902751af88ca-istio-data\") pod \"1febda0d-e4ac-436c-863e-902751af88ca\" (UID: \"1febda0d-e4ac-436c-863e-902751af88ca\") "
Nov 03 09:13:42.792100 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:42.792018    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1febda0d-e4ac-436c-863e-902751af88ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1febda0d-e4ac-436c-863e-902751af88ca-istio-podinfo\") pod \"1febda0d-e4ac-436c-863e-902751af88ca\" (UID: \"1febda0d-e4ac-436c-863e-902751af88ca\") "
Nov 03 09:13:42.792100 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:42.792047    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1febda0d-e4ac-436c-863e-902751af88ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1febda0d-e4ac-436c-863e-902751af88ca-workload-socket\") pod \"1febda0d-e4ac-436c-863e-902751af88ca\" (UID: \"1febda0d-e4ac-436c-863e-902751af88ca\") "
Nov 03 09:13:42.792100 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:42.792080    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1febda0d-e4ac-436c-863e-902751af88ca-istio-token\" (UniqueName: \"kubernetes.io/projected/1febda0d-e4ac-436c-863e-902751af88ca-istio-token\") pod \"1febda0d-e4ac-436c-863e-902751af88ca\" (UID: \"1febda0d-e4ac-436c-863e-902751af88ca\") "
Nov 03 09:13:42.792422 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:42.792129    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1febda0d-e4ac-436c-863e-902751af88ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1febda0d-e4ac-436c-863e-902751af88ca-credential-socket\") pod \"1febda0d-e4ac-436c-863e-902751af88ca\" (UID: \"1febda0d-e4ac-436c-863e-902751af88ca\") "
Nov 03 09:13:42.792422 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:42.792221    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1febda0d-e4ac-436c-863e-902751af88ca-workload-certs" pod "1febda0d-e4ac-436c-863e-902751af88ca" (UID: "1febda0d-e4ac-436c-863e-902751af88ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:13:42.792422 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:42.792334    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1febda0d-e4ac-436c-863e-902751af88ca-istiod-ca-cert" pod "1febda0d-e4ac-436c-863e-902751af88ca" (UID: "1febda0d-e4ac-436c-863e-902751af88ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:13:42.792422 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:42.792362    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1febda0d-e4ac-436c-863e-902751af88ca-workload-socket" pod "1febda0d-e4ac-436c-863e-902751af88ca" (UID: "1febda0d-e4ac-436c-863e-902751af88ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:13:42.792806 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:42.792531    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1febda0d-e4ac-436c-863e-902751af88ca-istio-data" pod "1febda0d-e4ac-436c-863e-902751af88ca" (UID: "1febda0d-e4ac-436c-863e-902751af88ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:13:42.792806 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:42.792588    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1febda0d-e4ac-436c-863e-902751af88ca-credential-socket" pod "1febda0d-e4ac-436c-863e-902751af88ca" (UID: "1febda0d-e4ac-436c-863e-902751af88ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:13:42.793016 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:42.792993    4473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1febda0d-e4ac-436c-863e-902751af88ca-istiod-ca-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:13:42.793117 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:42.793022    4473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1febda0d-e4ac-436c-863e-902751af88ca-istio-data\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:13:42.793117 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:42.793037    4473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1febda0d-e4ac-436c-863e-902751af88ca-workload-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:13:42.793117 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:42.793052    4473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1febda0d-e4ac-436c-863e-902751af88ca-credential-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:13:42.793117 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:42.793067    4473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1febda0d-e4ac-436c-863e-902751af88ca-workload-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:13:42.794307 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:42.794284    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1febda0d-e4ac-436c-863e-902751af88ca-istio-envoy" pod "1febda0d-e4ac-436c-863e-902751af88ca" (UID: "1febda0d-e4ac-436c-863e-902751af88ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:13:42.794409 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:42.794386    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1febda0d-e4ac-436c-863e-902751af88ca-kube-api-access-9vqbx" pod "1febda0d-e4ac-436c-863e-902751af88ca" (UID: "1febda0d-e4ac-436c-863e-902751af88ca"). InnerVolumeSpecName "kube-api-access-9vqbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:13:42.794462 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:42.794402    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1febda0d-e4ac-436c-863e-902751af88ca-istio-podinfo" pod "1febda0d-e4ac-436c-863e-902751af88ca" (UID: "1febda0d-e4ac-436c-863e-902751af88ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 09:13:42.794555 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:42.794543    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1febda0d-e4ac-436c-863e-902751af88ca-istio-token" pod "1febda0d-e4ac-436c-863e-902751af88ca" (UID: "1febda0d-e4ac-436c-863e-902751af88ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:13:42.893761 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:42.893732    4473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1febda0d-e4ac-436c-863e-902751af88ca-istio-podinfo\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:13:42.893761 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:42.893756    4473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1febda0d-e4ac-436c-863e-902751af88ca-istio-token\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:13:42.893761 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:42.893766    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vqbx\" (UniqueName: \"kubernetes.io/projected/1febda0d-e4ac-436c-863e-902751af88ca-kube-api-access-9vqbx\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:13:42.894092 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:42.893775    4473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1febda0d-e4ac-436c-863e-902751af88ca-istio-envoy\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:13:43.594342 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:43.594314    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"
Nov 03 09:13:43.594778 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:43.594348    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4" event={"ID":"1febda0d-e4ac-436c-863e-902751af88ca","Type":"ContainerDied","Data":"c05ad90193fed68f4013b3f4b1d6bd6ad9415d051c536e35ac9250bf20ee45c9"}
Nov 03 09:13:43.594778 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:43.594409    4473 scope.go:122] "RemoveContainer" containerID="171c40d5e73be093a52718481f53fce86703c829f1475be01709f718c6cef14d"
Nov 03 09:13:43.763715 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:43.763690    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"]
Nov 03 09:13:43.813917 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:43.813885    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-m7dm4"]
Nov 03 09:13:43.873916 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:43.873880    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1febda0d-e4ac-436c-863e-902751af88ca" path="/var/lib/kubelet/pods/1febda0d-e4ac-436c-863e-902751af88ca/volumes"
Nov 03 09:13:46.563214 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:46.563184    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-04934e-548676699b-sctv8_d59877f4-7a69-496e-b04b-7c3df4c9a2c4/payload-processing/0.log"
Nov 03 09:13:46.938049 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:13:46.938010    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:13:56.468478 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:56.468447    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"]
Nov 03 09:13:56.469026 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:56.468629    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b" podUID="e9252f6a-e0b5-45e4-8625-cc1af53d61c2" containerName="istio-proxy" containerID="cri-o://59e5d9ef838089cb0443567129743b58489774ce2b452573c2b781fc13848a82" gracePeriod=30
Nov 03 09:13:58.916760 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:58.916729    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8m98b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 09:13:58.917169 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:13:58.916787    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b" podUID="e9252f6a-e0b5-45e4-8625-cc1af53d61c2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 09:14:01.733718 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:01.733693    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"
Nov 03 09:14:01.752787 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:01.752724    4473 generic.go:356] "Generic (PLEG): container finished" podID="e9252f6a-e0b5-45e4-8625-cc1af53d61c2" containerID="59e5d9ef838089cb0443567129743b58489774ce2b452573c2b781fc13848a82" exitCode=0
Nov 03 09:14:01.752787 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:01.752768    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"
Nov 03 09:14:01.752976 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:01.752805    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b" event={"ID":"e9252f6a-e0b5-45e4-8625-cc1af53d61c2","Type":"ContainerDied","Data":"59e5d9ef838089cb0443567129743b58489774ce2b452573c2b781fc13848a82"}
Nov 03 09:14:01.752976 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:01.752858    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b" event={"ID":"e9252f6a-e0b5-45e4-8625-cc1af53d61c2","Type":"ContainerDied","Data":"f899c9086f44388ac20ac2ff7bafb221b40a6ffd667fbd1dd93378e67c7d6c82"}
Nov 03 09:14:01.752976 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:01.752884    4473 scope.go:122] "RemoveContainer" containerID="59e5d9ef838089cb0443567129743b58489774ce2b452573c2b781fc13848a82"
Nov 03 09:14:01.762570 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:01.762553    4473 scope.go:122] "RemoveContainer" containerID="59e5d9ef838089cb0443567129743b58489774ce2b452573c2b781fc13848a82"
Nov 03 09:14:01.762880 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:14:01.762858    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59e5d9ef838089cb0443567129743b58489774ce2b452573c2b781fc13848a82\": container with ID starting with 59e5d9ef838089cb0443567129743b58489774ce2b452573c2b781fc13848a82 not found: ID does not exist" containerID="59e5d9ef838089cb0443567129743b58489774ce2b452573c2b781fc13848a82"
Nov 03 09:14:01.763004 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:01.762885    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59e5d9ef838089cb0443567129743b58489774ce2b452573c2b781fc13848a82"} err="failed to get container status \"59e5d9ef838089cb0443567129743b58489774ce2b452573c2b781fc13848a82\": rpc error: code = NotFound desc = could not find container \"59e5d9ef838089cb0443567129743b58489774ce2b452573c2b781fc13848a82\": container with ID starting with 59e5d9ef838089cb0443567129743b58489774ce2b452573c2b781fc13848a82 not found: ID does not exist"
Nov 03 09:14:01.847335 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:01.847307    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-kube-api-access-dm5jz\" (UniqueName: \"kubernetes.io/projected/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-kube-api-access-dm5jz\") pod \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\" (UID: \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\") "
Nov 03 09:14:01.847615 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:01.847365    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-workload-certs\") pod \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\" (UID: \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\") "
Nov 03 09:14:01.847615 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:01.847409    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-workload-socket\") pod \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\" (UID: \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\") "
Nov 03 09:14:01.847615 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:01.847485    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-istiod-ca-cert\") pod \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\" (UID: \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\") "
Nov 03 09:14:01.847615 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:01.847529    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-istio-token\" (UniqueName: \"kubernetes.io/projected/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-istio-token\") pod \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\" (UID: \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\") "
Nov 03 09:14:01.847615 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:01.847572    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-credential-socket\") pod \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\" (UID: \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\") "
Nov 03 09:14:01.847615 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:01.847615    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-istio-data\") pod \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\" (UID: \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\") "
Nov 03 09:14:01.847995 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:01.847687    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-istio-envoy\") pod \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\" (UID: \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\") "
Nov 03 09:14:01.847995 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:01.847741    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-istio-podinfo\") pod \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\" (UID: \"e9252f6a-e0b5-45e4-8625-cc1af53d61c2\") "
Nov 03 09:14:01.848124 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:01.847999    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-istiod-ca-cert" pod "e9252f6a-e0b5-45e4-8625-cc1af53d61c2" (UID: "e9252f6a-e0b5-45e4-8625-cc1af53d61c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:14:01.848124 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:01.848080    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-istio-data" pod "e9252f6a-e0b5-45e4-8625-cc1af53d61c2" (UID: "e9252f6a-e0b5-45e4-8625-cc1af53d61c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:14:01.848342 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:01.848317    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-workload-certs" pod "e9252f6a-e0b5-45e4-8625-cc1af53d61c2" (UID: "e9252f6a-e0b5-45e4-8625-cc1af53d61c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:14:01.848438 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:01.848363    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-workload-socket" pod "e9252f6a-e0b5-45e4-8625-cc1af53d61c2" (UID: "e9252f6a-e0b5-45e4-8625-cc1af53d61c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:14:01.848438 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:01.848417    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-credential-socket" pod "e9252f6a-e0b5-45e4-8625-cc1af53d61c2" (UID: "e9252f6a-e0b5-45e4-8625-cc1af53d61c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:14:01.848551 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:01.848474    4473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-workload-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:14:01.848551 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:01.848496    4473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-istiod-ca-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:14:01.848551 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:01.848509    4473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-istio-data\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:14:01.849922 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:01.849893    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-kube-api-access-dm5jz" pod "e9252f6a-e0b5-45e4-8625-cc1af53d61c2" (UID: "e9252f6a-e0b5-45e4-8625-cc1af53d61c2"). InnerVolumeSpecName "kube-api-access-dm5jz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:14:01.850159 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:01.850136    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-istio-token" pod "e9252f6a-e0b5-45e4-8625-cc1af53d61c2" (UID: "e9252f6a-e0b5-45e4-8625-cc1af53d61c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:14:01.850339 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:01.850322    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-istio-podinfo" pod "e9252f6a-e0b5-45e4-8625-cc1af53d61c2" (UID: "e9252f6a-e0b5-45e4-8625-cc1af53d61c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 09:14:01.850938 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:01.850916    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-istio-envoy" pod "e9252f6a-e0b5-45e4-8625-cc1af53d61c2" (UID: "e9252f6a-e0b5-45e4-8625-cc1af53d61c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:14:01.950031 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:01.950002    4473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-istio-token\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:14:01.950031 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:01.950029    4473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-credential-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:14:01.950031 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:01.950039    4473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-istio-envoy\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:14:01.950418 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:01.950054    4473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-istio-podinfo\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:14:01.950418 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:01.950065    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm5jz\" (UniqueName: \"kubernetes.io/projected/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-kube-api-access-dm5jz\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:14:01.950418 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:01.950074    4473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9252f6a-e0b5-45e4-8625-cc1af53d61c2-workload-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:14:02.171653 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:02.171617    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"]
Nov 03 09:14:02.177458 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:02.177429    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8m98b"]
Nov 03 09:14:03.869809 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:03.869777    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9252f6a-e0b5-45e4-8625-cc1af53d61c2" path="/var/lib/kubelet/pods/e9252f6a-e0b5-45e4-8625-cc1af53d61c2/volumes"
Nov 03 09:14:04.685191 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:04.685159    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b2ae518-9f65r"]
Nov 03 09:14:04.685718 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:04.685704    4473 cpu_manager.go:414] "containerMap: removing container" podUID="1febda0d-e4ac-436c-863e-902751af88ca" containerName="istio-proxy"
Nov 03 09:14:04.685761 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:04.685720    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1febda0d-e4ac-436c-863e-902751af88ca" containerName="istio-proxy"
Nov 03 09:14:04.685761 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:04.685740    4473 cpu_manager.go:414] "containerMap: removing container" podUID="7213a5c7-1610-4205-8d73-3d096c44c277" containerName="main"
Nov 03 09:14:04.685761 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:04.685746    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7213a5c7-1610-4205-8d73-3d096c44c277" containerName="main"
Nov 03 09:14:04.685858 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:04.685766    4473 cpu_manager.go:414] "containerMap: removing container" podUID="ba3d1e4b-5057-429c-9c89-0045b964f1df" containerName="payload-pre-processing"
Nov 03 09:14:04.685858 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:04.685772    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba3d1e4b-5057-429c-9c89-0045b964f1df" containerName="payload-pre-processing"
Nov 03 09:14:04.685858 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:04.685792    4473 cpu_manager.go:414] "containerMap: removing container" podUID="e9252f6a-e0b5-45e4-8625-cc1af53d61c2" containerName="istio-proxy"
Nov 03 09:14:04.685858 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:04.685797    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9252f6a-e0b5-45e4-8625-cc1af53d61c2" containerName="istio-proxy"
Nov 03 09:14:04.685858 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:04.685805    4473 cpu_manager.go:414] "containerMap: removing container" podUID="048badaf-8a93-4e51-8308-2a75a72487f3" containerName="payload-pre-processing"
Nov 03 09:14:04.685858 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:04.685810    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="048badaf-8a93-4e51-8308-2a75a72487f3" containerName="payload-pre-processing"
Nov 03 09:14:04.686059 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:04.685881    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba3d1e4b-5057-429c-9c89-0045b964f1df" containerName="payload-pre-processing"
Nov 03 09:14:04.686059 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:04.685891    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9252f6a-e0b5-45e4-8625-cc1af53d61c2" containerName="istio-proxy"
Nov 03 09:14:04.686059 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:04.685897    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="048badaf-8a93-4e51-8308-2a75a72487f3" containerName="payload-pre-processing"
Nov 03 09:14:04.686059 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:04.685902    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="1febda0d-e4ac-436c-863e-902751af88ca" containerName="istio-proxy"
Nov 03 09:14:04.686059 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:04.685910    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="7213a5c7-1610-4205-8d73-3d096c44c277" containerName="main"
Nov 03 09:14:04.690462 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:04.690444    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2ae518-9f65r"
Nov 03 09:14:04.749069 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:04.749032    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:14:04.749069 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:04.749046    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:14:04.749069 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:04.749040    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:14:04.781747 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:04.781716    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d5440da-6d29-4e89-bfac-fd78fffed693-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b2ae518-9f65r\" (UID: \"9d5440da-6d29-4e89-bfac-fd78fffed693\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2ae518-9f65r"
Nov 03 09:14:04.781892 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:04.781754    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d5440da-6d29-4e89-bfac-fd78fffed693-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b2ae518-9f65r\" (UID: \"9d5440da-6d29-4e89-bfac-fd78fffed693\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2ae518-9f65r"
Nov 03 09:14:04.781892 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:04.781870    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gr5b\" (UniqueName: \"kubernetes.io/projected/9d5440da-6d29-4e89-bfac-fd78fffed693-kube-api-access-2gr5b\") pod \"maas-api-revoke-sub-3b2ae518-9f65r\" (UID: \"9d5440da-6d29-4e89-bfac-fd78fffed693\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2ae518-9f65r"
Nov 03 09:14:04.799244 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:04.799213    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:14:04.802431 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:04.802409    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b2ae518-9f65r"]
Nov 03 09:14:04.883301 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:04.883276    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gr5b\" (UniqueName: \"kubernetes.io/projected/9d5440da-6d29-4e89-bfac-fd78fffed693-kube-api-access-2gr5b\") pod \"maas-api-revoke-sub-3b2ae518-9f65r\" (UID: \"9d5440da-6d29-4e89-bfac-fd78fffed693\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2ae518-9f65r"
Nov 03 09:14:04.883693 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:04.883430    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d5440da-6d29-4e89-bfac-fd78fffed693-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b2ae518-9f65r\" (UID: \"9d5440da-6d29-4e89-bfac-fd78fffed693\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2ae518-9f65r"
Nov 03 09:14:04.883693 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:04.883455    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d5440da-6d29-4e89-bfac-fd78fffed693-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b2ae518-9f65r\" (UID: \"9d5440da-6d29-4e89-bfac-fd78fffed693\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2ae518-9f65r"
Nov 03 09:14:04.884195 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:04.884174    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d5440da-6d29-4e89-bfac-fd78fffed693-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b2ae518-9f65r\" (UID: \"9d5440da-6d29-4e89-bfac-fd78fffed693\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2ae518-9f65r"
Nov 03 09:14:04.885880 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:04.885853    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d5440da-6d29-4e89-bfac-fd78fffed693-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b2ae518-9f65r\" (UID: \"9d5440da-6d29-4e89-bfac-fd78fffed693\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2ae518-9f65r"
Nov 03 09:14:04.942209 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:04.942149    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gr5b\" (UniqueName: \"kubernetes.io/projected/9d5440da-6d29-4e89-bfac-fd78fffed693-kube-api-access-2gr5b\") pod \"maas-api-revoke-sub-3b2ae518-9f65r\" (UID: \"9d5440da-6d29-4e89-bfac-fd78fffed693\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2ae518-9f65r"
Nov 03 09:14:05.004096 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:05.004072    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2ae518-9f65r"
Nov 03 09:14:05.377858 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:05.377833    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b2ae518-9f65r"]
Nov 03 09:14:05.378695 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:14:05.378672    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d5440da_6d29_4e89_bfac_fd78fffed693.slice/crio-737643c7b546b5ad747fac8dce446b98abe92ce64eba3ca5b4b042e467cf18ec WatchSource:0}: Error finding container 737643c7b546b5ad747fac8dce446b98abe92ce64eba3ca5b4b042e467cf18ec: Status 404 returned error can't find the container with id 737643c7b546b5ad747fac8dce446b98abe92ce64eba3ca5b4b042e467cf18ec
Nov 03 09:14:05.777788 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:05.777752    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2ae518-9f65r" event={"ID":"9d5440da-6d29-4e89-bfac-fd78fffed693","Type":"ContainerStarted","Data":"c9bc9a74c085b680eecc2729927b9d536ace672daf2c2d48b79d618410005c54"}
Nov 03 09:14:05.777974 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:05.777814    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2ae518-9f65r" event={"ID":"9d5440da-6d29-4e89-bfac-fd78fffed693","Type":"ContainerStarted","Data":"737643c7b546b5ad747fac8dce446b98abe92ce64eba3ca5b4b042e467cf18ec"}
Nov 03 09:14:05.895580 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:05.895434    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2ae518-9f65r" podStartSLOduration=1.895415758 podStartE2EDuration="1.895415758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:14:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:14:05.893565774 +0000 UTC m=+2070.162629209" watchObservedRunningTime="2026-11-03 09:14:05.895415758 +0000 UTC m=+2070.164479193"
Nov 03 09:14:06.785160 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:06.785125    4473 generic.go:356] "Generic (PLEG): container finished" podID="9d5440da-6d29-4e89-bfac-fd78fffed693" containerID="c9bc9a74c085b680eecc2729927b9d536ace672daf2c2d48b79d618410005c54" exitCode=0
Nov 03 09:14:06.785351 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:06.785205    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2ae518-9f65r" event={"ID":"9d5440da-6d29-4e89-bfac-fd78fffed693","Type":"ContainerDied","Data":"c9bc9a74c085b680eecc2729927b9d536ace672daf2c2d48b79d618410005c54"}
Nov 03 09:14:07.933017 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:07.932994    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2ae518-9f65r"
Nov 03 09:14:08.026841 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:08.026801    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d5440da-6d29-4e89-bfac-fd78fffed693-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d5440da-6d29-4e89-bfac-fd78fffed693-maas-api-openshift-ca\") pod \"9d5440da-6d29-4e89-bfac-fd78fffed693\" (UID: \"9d5440da-6d29-4e89-bfac-fd78fffed693\") "
Nov 03 09:14:08.027052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:08.026859    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d5440da-6d29-4e89-bfac-fd78fffed693-kube-api-access-2gr5b\" (UniqueName: \"kubernetes.io/projected/9d5440da-6d29-4e89-bfac-fd78fffed693-kube-api-access-2gr5b\") pod \"9d5440da-6d29-4e89-bfac-fd78fffed693\" (UID: \"9d5440da-6d29-4e89-bfac-fd78fffed693\") "
Nov 03 09:14:08.027052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:08.027030    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d5440da-6d29-4e89-bfac-fd78fffed693-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d5440da-6d29-4e89-bfac-fd78fffed693-maas-api-xks-ca\") pod \"9d5440da-6d29-4e89-bfac-fd78fffed693\" (UID: \"9d5440da-6d29-4e89-bfac-fd78fffed693\") "
Nov 03 09:14:08.027204 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:08.027182    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d5440da-6d29-4e89-bfac-fd78fffed693-maas-api-openshift-ca" pod "9d5440da-6d29-4e89-bfac-fd78fffed693" (UID: "9d5440da-6d29-4e89-bfac-fd78fffed693"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:14:08.027506 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:08.027469    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d5440da-6d29-4e89-bfac-fd78fffed693-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:14:08.029216 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:08.029191    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d5440da-6d29-4e89-bfac-fd78fffed693-kube-api-access-2gr5b" pod "9d5440da-6d29-4e89-bfac-fd78fffed693" (UID: "9d5440da-6d29-4e89-bfac-fd78fffed693"). InnerVolumeSpecName "kube-api-access-2gr5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:14:08.029216 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:08.029213    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d5440da-6d29-4e89-bfac-fd78fffed693-maas-api-xks-ca" pod "9d5440da-6d29-4e89-bfac-fd78fffed693" (UID: "9d5440da-6d29-4e89-bfac-fd78fffed693"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:14:08.128814 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:08.128784    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d5440da-6d29-4e89-bfac-fd78fffed693-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:14:08.128814 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:08.128814    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gr5b\" (UniqueName: \"kubernetes.io/projected/9d5440da-6d29-4e89-bfac-fd78fffed693-kube-api-access-2gr5b\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:14:08.803624 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:08.803575    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2ae518-9f65r" event={"ID":"9d5440da-6d29-4e89-bfac-fd78fffed693","Type":"ContainerDied","Data":"737643c7b546b5ad747fac8dce446b98abe92ce64eba3ca5b4b042e467cf18ec"}
Nov 03 09:14:08.803829 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:08.803657    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="737643c7b546b5ad747fac8dce446b98abe92ce64eba3ca5b4b042e467cf18ec"
Nov 03 09:14:08.803829 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:08.803604    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b2ae518-9f65r"
Nov 03 09:14:09.407050 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:09.407014    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:14:16.937495 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:14:16.937446    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:14:20.273798 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:20.273765    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"]
Nov 03 09:14:20.274629 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:20.274595    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km" podUID="c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3" containerName="istio-proxy" containerID="cri-o://0b0d3d0082a5ecd43f2e97370b1f448705f8f924998f53d180ceb25eae18d314" gracePeriod=30
Nov 03 09:14:22.652418 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:22.652381    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"]
Nov 03 09:14:22.652906 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:22.652636    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th" podUID="39799a31-d7b7-4948-af12-23cb1f426fc5" containerName="istio-proxy" containerID="cri-o://f04fbc63937a5fcdc5a588aebc555a2b4220974320da951445eb70979cf76061" gracePeriod=30
Nov 03 09:14:23.126962 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:23.126915    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-107e604c-rn4sh"]
Nov 03 09:14:23.127522 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:23.127502    4473 cpu_manager.go:414] "containerMap: removing container" podUID="9d5440da-6d29-4e89-bfac-fd78fffed693" containerName="revoke-keys"
Nov 03 09:14:23.127628 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:23.127524    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d5440da-6d29-4e89-bfac-fd78fffed693" containerName="revoke-keys"
Nov 03 09:14:23.127685 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:23.127652    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d5440da-6d29-4e89-bfac-fd78fffed693" containerName="revoke-keys"
Nov 03 09:14:23.130974 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:23.130936    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-107e604c-rn4sh"
Nov 03 09:14:23.149808 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:23.149784    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a3260e-cf90-4fa4-883a-10894bb35df0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-107e604c-rn4sh\" (UID: \"f4a3260e-cf90-4fa4-883a-10894bb35df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107e604c-rn4sh"
Nov 03 09:14:23.149939 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:23.149852    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a3260e-cf90-4fa4-883a-10894bb35df0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-107e604c-rn4sh\" (UID: \"f4a3260e-cf90-4fa4-883a-10894bb35df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107e604c-rn4sh"
Nov 03 09:14:23.150033 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:23.150012    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7jkx\" (UniqueName: \"kubernetes.io/projected/f4a3260e-cf90-4fa4-883a-10894bb35df0-kube-api-access-s7jkx\") pod \"maas-api-revoke-sub-107e604c-rn4sh\" (UID: \"f4a3260e-cf90-4fa4-883a-10894bb35df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107e604c-rn4sh"
Nov 03 09:14:23.187715 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:23.187370    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:14:23.187715 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:23.187494    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:14:23.190805 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:23.190782    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-107e604c-rn4sh"]
Nov 03 09:14:23.222441 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:23.222417    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:14:23.236450 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:23.236423    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:14:23.251028 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:23.250999    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a3260e-cf90-4fa4-883a-10894bb35df0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-107e604c-rn4sh\" (UID: \"f4a3260e-cf90-4fa4-883a-10894bb35df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107e604c-rn4sh"
Nov 03 09:14:23.251149 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:23.251050    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a3260e-cf90-4fa4-883a-10894bb35df0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-107e604c-rn4sh\" (UID: \"f4a3260e-cf90-4fa4-883a-10894bb35df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107e604c-rn4sh"
Nov 03 09:14:23.251149 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:23.251098    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7jkx\" (UniqueName: \"kubernetes.io/projected/f4a3260e-cf90-4fa4-883a-10894bb35df0-kube-api-access-s7jkx\") pod \"maas-api-revoke-sub-107e604c-rn4sh\" (UID: \"f4a3260e-cf90-4fa4-883a-10894bb35df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107e604c-rn4sh"
Nov 03 09:14:23.251837 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:23.251808    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a3260e-cf90-4fa4-883a-10894bb35df0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-107e604c-rn4sh\" (UID: \"f4a3260e-cf90-4fa4-883a-10894bb35df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107e604c-rn4sh"
Nov 03 09:14:23.254255 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:23.254231    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a3260e-cf90-4fa4-883a-10894bb35df0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-107e604c-rn4sh\" (UID: \"f4a3260e-cf90-4fa4-883a-10894bb35df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107e604c-rn4sh"
Nov 03 09:14:23.311362 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:23.311341    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7jkx\" (UniqueName: \"kubernetes.io/projected/f4a3260e-cf90-4fa4-883a-10894bb35df0-kube-api-access-s7jkx\") pod \"maas-api-revoke-sub-107e604c-rn4sh\" (UID: \"f4a3260e-cf90-4fa4-883a-10894bb35df0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107e604c-rn4sh"
Nov 03 09:14:23.444215 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:23.444132    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-107e604c-rn4sh"
Nov 03 09:14:23.831259 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:23.831230    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-107e604c-rn4sh"]
Nov 03 09:14:23.831774 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:14:23.831749    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4a3260e_cf90_4fa4_883a_10894bb35df0.slice/crio-9959c2ef821b239862c661d35637676db2698567df896cfc26707c0d6b1ca7aa WatchSource:0}: Error finding container 9959c2ef821b239862c661d35637676db2698567df896cfc26707c0d6b1ca7aa: Status 404 returned error can't find the container with id 9959c2ef821b239862c661d35637676db2698567df896cfc26707c0d6b1ca7aa
Nov 03 09:14:23.941839 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:23.941810    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-107e604c-rn4sh" event={"ID":"f4a3260e-cf90-4fa4-883a-10894bb35df0","Type":"ContainerStarted","Data":"9959c2ef821b239862c661d35637676db2698567df896cfc26707c0d6b1ca7aa"}
Nov 03 09:14:24.285993 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:24.285938    4473 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 09:14:24.286142 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:24.286010    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th" podUID="39799a31-d7b7-4948-af12-23cb1f426fc5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 09:14:24.949267 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:24.949176    4473 generic.go:356] "Generic (PLEG): container finished" podID="f4a3260e-cf90-4fa4-883a-10894bb35df0" containerID="0058eb1144a3fd2dbfb6a2445140d306cf409fd675fc83bf2c44d710c93f139e" exitCode=0
Nov 03 09:14:24.949705 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:24.949261    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-107e604c-rn4sh" event={"ID":"f4a3260e-cf90-4fa4-883a-10894bb35df0","Type":"ContainerDied","Data":"0058eb1144a3fd2dbfb6a2445140d306cf409fd675fc83bf2c44d710c93f139e"}
Nov 03 09:14:25.529256 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:25.529230    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"
Nov 03 09:14:25.577439 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:25.577369    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-istio-envoy\") pod \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\" (UID: \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\") "
Nov 03 09:14:25.577593 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:25.577446    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-credential-socket\") pod \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\" (UID: \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\") "
Nov 03 09:14:25.577593 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:25.577503    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-kube-api-access-ppn75\" (UniqueName: \"kubernetes.io/projected/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-kube-api-access-ppn75\") pod \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\" (UID: \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\") "
Nov 03 09:14:25.577751 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:25.577611    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-istio-data\") pod \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\" (UID: \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\") "
Nov 03 09:14:25.577751 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:25.577738    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-workload-socket\") pod \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\" (UID: \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\") "
Nov 03 09:14:25.577852 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:25.577782    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-istio-token\" (UniqueName: \"kubernetes.io/projected/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-istio-token\") pod \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\" (UID: \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\") "
Nov 03 09:14:25.577852 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:25.577779    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-credential-socket" pod "c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3" (UID: "c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:14:25.577852 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:25.577811    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-istio-podinfo\") pod \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\" (UID: \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\") "
Nov 03 09:14:25.577852 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:25.577827    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-istio-data" pod "c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3" (UID: "c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:14:25.578097 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:25.577933    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-istiod-ca-cert\") pod \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\" (UID: \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\") "
Nov 03 09:14:25.578097 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:25.577992    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-workload-socket" pod "c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3" (UID: "c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:14:25.578097 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:25.578058    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-workload-certs\") pod \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\" (UID: \"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3\") "
Nov 03 09:14:25.578596 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:25.578391    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-workload-certs" pod "c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3" (UID: "c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:14:25.578596 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:25.578482    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-istiod-ca-cert" pod "c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3" (UID: "c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:14:25.578812 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:25.578786    4473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-workload-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:14:25.578885 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:25.578815    4473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-istiod-ca-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:14:25.578885 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:25.578831    4473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-workload-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:14:25.578885 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:25.578847    4473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-credential-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:14:25.578885 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:25.578861    4473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-istio-data\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:14:25.579958 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:25.579916    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-istio-envoy" pod "c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3" (UID: "c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:14:25.580129 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:25.580101    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-istio-token" pod "c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3" (UID: "c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:14:25.580129 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:25.580113    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-istio-podinfo" pod "c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3" (UID: "c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 09:14:25.580129 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:25.580119    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-kube-api-access-ppn75" pod "c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3" (UID: "c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3"). InnerVolumeSpecName "kube-api-access-ppn75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:14:25.679912 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:25.679881    4473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-istio-envoy\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:14:25.679912 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:25.679910    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppn75\" (UniqueName: \"kubernetes.io/projected/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-kube-api-access-ppn75\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:14:25.680098 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:25.679922    4473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-istio-token\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:14:25.680098 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:25.679931    4473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3-istio-podinfo\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:14:25.956818 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:25.956782    4473 generic.go:356] "Generic (PLEG): container finished" podID="c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3" containerID="0b0d3d0082a5ecd43f2e97370b1f448705f8f924998f53d180ceb25eae18d314" exitCode=0
Nov 03 09:14:25.957274 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:25.956821    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"
Nov 03 09:14:25.957274 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:25.956851    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km" event={"ID":"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3","Type":"ContainerDied","Data":"0b0d3d0082a5ecd43f2e97370b1f448705f8f924998f53d180ceb25eae18d314"}
Nov 03 09:14:25.957274 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:25.956911    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km" event={"ID":"c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3","Type":"ContainerDied","Data":"d6f6a3f69251c9edf5943cf68f6de90f802b403ae0204a3fb46c90805f8d7387"}
Nov 03 09:14:25.957274 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:25.956933    4473 scope.go:122] "RemoveContainer" containerID="0b0d3d0082a5ecd43f2e97370b1f448705f8f924998f53d180ceb25eae18d314"
Nov 03 09:14:25.966780 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:25.966759    4473 scope.go:122] "RemoveContainer" containerID="0b0d3d0082a5ecd43f2e97370b1f448705f8f924998f53d180ceb25eae18d314"
Nov 03 09:14:25.967107 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:14:25.967079    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b0d3d0082a5ecd43f2e97370b1f448705f8f924998f53d180ceb25eae18d314\": container with ID starting with 0b0d3d0082a5ecd43f2e97370b1f448705f8f924998f53d180ceb25eae18d314 not found: ID does not exist" containerID="0b0d3d0082a5ecd43f2e97370b1f448705f8f924998f53d180ceb25eae18d314"
Nov 03 09:14:25.967177 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:25.967121    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b0d3d0082a5ecd43f2e97370b1f448705f8f924998f53d180ceb25eae18d314"} err="failed to get container status \"0b0d3d0082a5ecd43f2e97370b1f448705f8f924998f53d180ceb25eae18d314\": rpc error: code = NotFound desc = could not find container \"0b0d3d0082a5ecd43f2e97370b1f448705f8f924998f53d180ceb25eae18d314\": container with ID starting with 0b0d3d0082a5ecd43f2e97370b1f448705f8f924998f53d180ceb25eae18d314 not found: ID does not exist"
Nov 03 09:14:26.090090 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:26.090063    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-107e604c-rn4sh"
Nov 03 09:14:26.127915 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:26.127844    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"]
Nov 03 09:14:26.178488 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:26.178462    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-18f9da-openshift-default-5686c956bc-nm4km"]
Nov 03 09:14:26.186194 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:26.186167    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4a3260e-cf90-4fa4-883a-10894bb35df0-kube-api-access-s7jkx\" (UniqueName: \"kubernetes.io/projected/f4a3260e-cf90-4fa4-883a-10894bb35df0-kube-api-access-s7jkx\") pod \"f4a3260e-cf90-4fa4-883a-10894bb35df0\" (UID: \"f4a3260e-cf90-4fa4-883a-10894bb35df0\") "
Nov 03 09:14:26.186332 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:26.186315    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4a3260e-cf90-4fa4-883a-10894bb35df0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a3260e-cf90-4fa4-883a-10894bb35df0-maas-api-openshift-ca\") pod \"f4a3260e-cf90-4fa4-883a-10894bb35df0\" (UID: \"f4a3260e-cf90-4fa4-883a-10894bb35df0\") "
Nov 03 09:14:26.186453 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:26.186435    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4a3260e-cf90-4fa4-883a-10894bb35df0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a3260e-cf90-4fa4-883a-10894bb35df0-maas-api-xks-ca\") pod \"f4a3260e-cf90-4fa4-883a-10894bb35df0\" (UID: \"f4a3260e-cf90-4fa4-883a-10894bb35df0\") "
Nov 03 09:14:26.186688 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:26.186657    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4a3260e-cf90-4fa4-883a-10894bb35df0-maas-api-openshift-ca" pod "f4a3260e-cf90-4fa4-883a-10894bb35df0" (UID: "f4a3260e-cf90-4fa4-883a-10894bb35df0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:14:26.186806 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:26.186791    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a3260e-cf90-4fa4-883a-10894bb35df0-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:14:26.188279 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:26.188258    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4a3260e-cf90-4fa4-883a-10894bb35df0-kube-api-access-s7jkx" pod "f4a3260e-cf90-4fa4-883a-10894bb35df0" (UID: "f4a3260e-cf90-4fa4-883a-10894bb35df0"). InnerVolumeSpecName "kube-api-access-s7jkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:14:26.188554 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:26.188538    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4a3260e-cf90-4fa4-883a-10894bb35df0-maas-api-xks-ca" pod "f4a3260e-cf90-4fa4-883a-10894bb35df0" (UID: "f4a3260e-cf90-4fa4-883a-10894bb35df0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:14:26.287811 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:26.287784    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a3260e-cf90-4fa4-883a-10894bb35df0-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:14:26.287811 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:26.287809    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7jkx\" (UniqueName: \"kubernetes.io/projected/f4a3260e-cf90-4fa4-883a-10894bb35df0-kube-api-access-s7jkx\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:14:26.963853 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:26.963828    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-107e604c-rn4sh"
Nov 03 09:14:26.964285 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:26.963845    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-107e604c-rn4sh" event={"ID":"f4a3260e-cf90-4fa4-883a-10894bb35df0","Type":"ContainerDied","Data":"9959c2ef821b239862c661d35637676db2698567df896cfc26707c0d6b1ca7aa"}
Nov 03 09:14:26.964285 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:26.963896    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9959c2ef821b239862c661d35637676db2698567df896cfc26707c0d6b1ca7aa"
Nov 03 09:14:27.869818 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:27.869790    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3" path="/var/lib/kubelet/pods/c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3/volumes"
Nov 03 09:14:27.924103 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:27.924052    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"
Nov 03 09:14:27.971010 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:27.970984    4473 generic.go:356] "Generic (PLEG): container finished" podID="39799a31-d7b7-4948-af12-23cb1f426fc5" containerID="f04fbc63937a5fcdc5a588aebc555a2b4220974320da951445eb70979cf76061" exitCode=0
Nov 03 09:14:27.971356 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:27.971029    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"
Nov 03 09:14:27.971356 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:27.971063    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th" event={"ID":"39799a31-d7b7-4948-af12-23cb1f426fc5","Type":"ContainerDied","Data":"f04fbc63937a5fcdc5a588aebc555a2b4220974320da951445eb70979cf76061"}
Nov 03 09:14:27.971356 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:27.971116    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th" event={"ID":"39799a31-d7b7-4948-af12-23cb1f426fc5","Type":"ContainerDied","Data":"593c8712c710dd0ba076eba2aef087dfc5c2b1c7f2f561657b9b5d5af2848c57"}
Nov 03 09:14:27.971356 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:27.971147    4473 scope.go:122] "RemoveContainer" containerID="f04fbc63937a5fcdc5a588aebc555a2b4220974320da951445eb70979cf76061"
Nov 03 09:14:27.980489 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:27.980474    4473 scope.go:122] "RemoveContainer" containerID="f04fbc63937a5fcdc5a588aebc555a2b4220974320da951445eb70979cf76061"
Nov 03 09:14:27.980762 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:14:27.980740    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f04fbc63937a5fcdc5a588aebc555a2b4220974320da951445eb70979cf76061\": container with ID starting with f04fbc63937a5fcdc5a588aebc555a2b4220974320da951445eb70979cf76061 not found: ID does not exist" containerID="f04fbc63937a5fcdc5a588aebc555a2b4220974320da951445eb70979cf76061"
Nov 03 09:14:27.980842 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:27.980765    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f04fbc63937a5fcdc5a588aebc555a2b4220974320da951445eb70979cf76061"} err="failed to get container status \"f04fbc63937a5fcdc5a588aebc555a2b4220974320da951445eb70979cf76061\": rpc error: code = NotFound desc = could not find container \"f04fbc63937a5fcdc5a588aebc555a2b4220974320da951445eb70979cf76061\": container with ID starting with f04fbc63937a5fcdc5a588aebc555a2b4220974320da951445eb70979cf76061 not found: ID does not exist"
Nov 03 09:14:28.007740 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:28.007719    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39799a31-d7b7-4948-af12-23cb1f426fc5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39799a31-d7b7-4948-af12-23cb1f426fc5-workload-certs\") pod \"39799a31-d7b7-4948-af12-23cb1f426fc5\" (UID: \"39799a31-d7b7-4948-af12-23cb1f426fc5\") "
Nov 03 09:14:28.007828 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:28.007770    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39799a31-d7b7-4948-af12-23cb1f426fc5-kube-api-access-94z2p\" (UniqueName: \"kubernetes.io/projected/39799a31-d7b7-4948-af12-23cb1f426fc5-kube-api-access-94z2p\") pod \"39799a31-d7b7-4948-af12-23cb1f426fc5\" (UID: \"39799a31-d7b7-4948-af12-23cb1f426fc5\") "
Nov 03 09:14:28.007828 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:28.007794    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39799a31-d7b7-4948-af12-23cb1f426fc5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39799a31-d7b7-4948-af12-23cb1f426fc5-workload-socket\") pod \"39799a31-d7b7-4948-af12-23cb1f426fc5\" (UID: \"39799a31-d7b7-4948-af12-23cb1f426fc5\") "
Nov 03 09:14:28.007899 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:28.007877    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39799a31-d7b7-4948-af12-23cb1f426fc5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39799a31-d7b7-4948-af12-23cb1f426fc5-istio-envoy\") pod \"39799a31-d7b7-4948-af12-23cb1f426fc5\" (UID: \"39799a31-d7b7-4948-af12-23cb1f426fc5\") "
Nov 03 09:14:28.007939 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:28.007922    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39799a31-d7b7-4948-af12-23cb1f426fc5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39799a31-d7b7-4948-af12-23cb1f426fc5-istio-podinfo\") pod \"39799a31-d7b7-4948-af12-23cb1f426fc5\" (UID: \"39799a31-d7b7-4948-af12-23cb1f426fc5\") "
Nov 03 09:14:28.008022 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:28.007937    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39799a31-d7b7-4948-af12-23cb1f426fc5-workload-certs" pod "39799a31-d7b7-4948-af12-23cb1f426fc5" (UID: "39799a31-d7b7-4948-af12-23cb1f426fc5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:14:28.008022 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:28.007977    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39799a31-d7b7-4948-af12-23cb1f426fc5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39799a31-d7b7-4948-af12-23cb1f426fc5-credential-socket\") pod \"39799a31-d7b7-4948-af12-23cb1f426fc5\" (UID: \"39799a31-d7b7-4948-af12-23cb1f426fc5\") "
Nov 03 09:14:28.008128 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:28.008049    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39799a31-d7b7-4948-af12-23cb1f426fc5-istio-token\" (UniqueName: \"kubernetes.io/projected/39799a31-d7b7-4948-af12-23cb1f426fc5-istio-token\") pod \"39799a31-d7b7-4948-af12-23cb1f426fc5\" (UID: \"39799a31-d7b7-4948-af12-23cb1f426fc5\") "
Nov 03 09:14:28.008184 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:28.008132    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39799a31-d7b7-4948-af12-23cb1f426fc5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39799a31-d7b7-4948-af12-23cb1f426fc5-istio-data\") pod \"39799a31-d7b7-4948-af12-23cb1f426fc5\" (UID: \"39799a31-d7b7-4948-af12-23cb1f426fc5\") "
Nov 03 09:14:28.008241 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:28.008184    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39799a31-d7b7-4948-af12-23cb1f426fc5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39799a31-d7b7-4948-af12-23cb1f426fc5-istiod-ca-cert\") pod \"39799a31-d7b7-4948-af12-23cb1f426fc5\" (UID: \"39799a31-d7b7-4948-af12-23cb1f426fc5\") "
Nov 03 09:14:28.008241 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:28.008211    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39799a31-d7b7-4948-af12-23cb1f426fc5-workload-socket" pod "39799a31-d7b7-4948-af12-23cb1f426fc5" (UID: "39799a31-d7b7-4948-af12-23cb1f426fc5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:14:28.008408 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:28.008382    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39799a31-d7b7-4948-af12-23cb1f426fc5-istio-data" pod "39799a31-d7b7-4948-af12-23cb1f426fc5" (UID: "39799a31-d7b7-4948-af12-23cb1f426fc5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:14:28.008542 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:28.008496    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39799a31-d7b7-4948-af12-23cb1f426fc5-credential-socket" pod "39799a31-d7b7-4948-af12-23cb1f426fc5" (UID: "39799a31-d7b7-4948-af12-23cb1f426fc5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:14:28.008836 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:28.008632    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39799a31-d7b7-4948-af12-23cb1f426fc5-istiod-ca-cert" pod "39799a31-d7b7-4948-af12-23cb1f426fc5" (UID: "39799a31-d7b7-4948-af12-23cb1f426fc5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:14:28.008836 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:28.008822    4473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39799a31-d7b7-4948-af12-23cb1f426fc5-istiod-ca-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:14:28.008997 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:28.008843    4473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39799a31-d7b7-4948-af12-23cb1f426fc5-workload-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:14:28.008997 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:28.008860    4473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39799a31-d7b7-4948-af12-23cb1f426fc5-workload-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:14:28.008997 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:28.008874    4473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39799a31-d7b7-4948-af12-23cb1f426fc5-credential-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:14:28.008997 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:28.008885    4473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39799a31-d7b7-4948-af12-23cb1f426fc5-istio-data\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:14:28.010285 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:28.010263    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39799a31-d7b7-4948-af12-23cb1f426fc5-istio-envoy" pod "39799a31-d7b7-4948-af12-23cb1f426fc5" (UID: "39799a31-d7b7-4948-af12-23cb1f426fc5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:14:28.010363 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:28.010286    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39799a31-d7b7-4948-af12-23cb1f426fc5-istio-token" pod "39799a31-d7b7-4948-af12-23cb1f426fc5" (UID: "39799a31-d7b7-4948-af12-23cb1f426fc5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:14:28.010363 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:28.010309    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39799a31-d7b7-4948-af12-23cb1f426fc5-kube-api-access-94z2p" pod "39799a31-d7b7-4948-af12-23cb1f426fc5" (UID: "39799a31-d7b7-4948-af12-23cb1f426fc5"). InnerVolumeSpecName "kube-api-access-94z2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:14:28.010439 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:28.010374    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39799a31-d7b7-4948-af12-23cb1f426fc5-istio-podinfo" pod "39799a31-d7b7-4948-af12-23cb1f426fc5" (UID: "39799a31-d7b7-4948-af12-23cb1f426fc5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 09:14:28.110512 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:28.110484    4473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39799a31-d7b7-4948-af12-23cb1f426fc5-istio-envoy\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:14:28.110512 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:28.110509    4473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39799a31-d7b7-4948-af12-23cb1f426fc5-istio-podinfo\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:14:28.110512 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:28.110518    4473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39799a31-d7b7-4948-af12-23cb1f426fc5-istio-token\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:14:28.110796 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:28.110534    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94z2p\" (UniqueName: \"kubernetes.io/projected/39799a31-d7b7-4948-af12-23cb1f426fc5-kube-api-access-94z2p\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:14:28.395379 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:28.395340    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"]
Nov 03 09:14:28.403732 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:28.403706    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-dq9th"]
Nov 03 09:14:29.869837 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:29.869808    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39799a31-d7b7-4948-af12-23cb1f426fc5" path="/var/lib/kubelet/pods/39799a31-d7b7-4948-af12-23cb1f426fc5/volumes"
Nov 03 09:14:36.302708 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:36.302679    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 09:14:36.303276 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:36.303254    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 09:14:40.465334 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:40.465299    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd5a7382-6z7sb"]
Nov 03 09:14:40.465814 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:40.465801    4473 cpu_manager.go:414] "containerMap: removing container" podUID="f4a3260e-cf90-4fa4-883a-10894bb35df0" containerName="revoke-keys"
Nov 03 09:14:40.465858 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:40.465816    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4a3260e-cf90-4fa4-883a-10894bb35df0" containerName="revoke-keys"
Nov 03 09:14:40.465858 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:40.465845    4473 cpu_manager.go:414] "containerMap: removing container" podUID="39799a31-d7b7-4948-af12-23cb1f426fc5" containerName="istio-proxy"
Nov 03 09:14:40.465858 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:40.465850    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39799a31-d7b7-4948-af12-23cb1f426fc5" containerName="istio-proxy"
Nov 03 09:14:40.465968 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:40.465862    4473 cpu_manager.go:414] "containerMap: removing container" podUID="c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3" containerName="istio-proxy"
Nov 03 09:14:40.465968 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:40.465868    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3" containerName="istio-proxy"
Nov 03 09:14:40.465968 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:40.465958    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="39799a31-d7b7-4948-af12-23cb1f426fc5" containerName="istio-proxy"
Nov 03 09:14:40.465968 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:40.465967    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1b72f6f-1e8e-4bb4-a8f6-16eb66cc7ba3" containerName="istio-proxy"
Nov 03 09:14:40.466092 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:40.465976    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4a3260e-cf90-4fa4-883a-10894bb35df0" containerName="revoke-keys"
Nov 03 09:14:40.470522 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:40.470507    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5a7382-6z7sb"
Nov 03 09:14:40.527114 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:40.527079    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd5a7382-6z7sb"]
Nov 03 09:14:40.573450 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:40.573416    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:14:40.573450 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:40.573444    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:14:40.573656 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:40.573462    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:14:40.574029 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:40.574004    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:14:40.574101 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:40.574057    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkkw6\" (UniqueName: \"kubernetes.io/projected/447303f1-c173-4810-acfd-51993e615c0f-kube-api-access-fkkw6\") pod \"maas-api-revoke-sub-bd5a7382-6z7sb\" (UID: \"447303f1-c173-4810-acfd-51993e615c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5a7382-6z7sb"
Nov 03 09:14:40.574101 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:40.574095    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/447303f1-c173-4810-acfd-51993e615c0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd5a7382-6z7sb\" (UID: \"447303f1-c173-4810-acfd-51993e615c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5a7382-6z7sb"
Nov 03 09:14:40.574204 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:40.574122    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/447303f1-c173-4810-acfd-51993e615c0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd5a7382-6z7sb\" (UID: \"447303f1-c173-4810-acfd-51993e615c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5a7382-6z7sb"
Nov 03 09:14:40.675697 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:40.675665    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkkw6\" (UniqueName: \"kubernetes.io/projected/447303f1-c173-4810-acfd-51993e615c0f-kube-api-access-fkkw6\") pod \"maas-api-revoke-sub-bd5a7382-6z7sb\" (UID: \"447303f1-c173-4810-acfd-51993e615c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5a7382-6z7sb"
Nov 03 09:14:40.675899 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:40.675718    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/447303f1-c173-4810-acfd-51993e615c0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd5a7382-6z7sb\" (UID: \"447303f1-c173-4810-acfd-51993e615c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5a7382-6z7sb"
Nov 03 09:14:40.675899 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:40.675766    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/447303f1-c173-4810-acfd-51993e615c0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd5a7382-6z7sb\" (UID: \"447303f1-c173-4810-acfd-51993e615c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5a7382-6z7sb"
Nov 03 09:14:40.676381 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:40.676344    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/447303f1-c173-4810-acfd-51993e615c0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd5a7382-6z7sb\" (UID: \"447303f1-c173-4810-acfd-51993e615c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5a7382-6z7sb"
Nov 03 09:14:40.678201 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:40.678181    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/447303f1-c173-4810-acfd-51993e615c0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd5a7382-6z7sb\" (UID: \"447303f1-c173-4810-acfd-51993e615c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5a7382-6z7sb"
Nov 03 09:14:40.734921 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:40.734845    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkkw6\" (UniqueName: \"kubernetes.io/projected/447303f1-c173-4810-acfd-51993e615c0f-kube-api-access-fkkw6\") pod \"maas-api-revoke-sub-bd5a7382-6z7sb\" (UID: \"447303f1-c173-4810-acfd-51993e615c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5a7382-6z7sb"
Nov 03 09:14:40.783397 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:40.783359    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5a7382-6z7sb"
Nov 03 09:14:41.180868 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:41.180838    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd5a7382-6z7sb"]
Nov 03 09:14:41.183028 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:14:41.183002    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod447303f1_c173_4810_acfd_51993e615c0f.slice/crio-2f4a1226f27bdffc66ec682d2ba875812c031054fc8de4da4beab84b71567166 WatchSource:0}: Error finding container 2f4a1226f27bdffc66ec682d2ba875812c031054fc8de4da4beab84b71567166: Status 404 returned error can't find the container with id 2f4a1226f27bdffc66ec682d2ba875812c031054fc8de4da4beab84b71567166
Nov 03 09:14:42.053182 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:42.053147    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5a7382-6z7sb" event={"ID":"447303f1-c173-4810-acfd-51993e615c0f","Type":"ContainerStarted","Data":"4ac96d55c1d9c2f443010b9736bb9d4835c983ff928fb8c1092a034145cc1929"}
Nov 03 09:14:42.053677 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:42.053197    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5a7382-6z7sb" event={"ID":"447303f1-c173-4810-acfd-51993e615c0f","Type":"ContainerStarted","Data":"2f4a1226f27bdffc66ec682d2ba875812c031054fc8de4da4beab84b71567166"}
Nov 03 09:14:42.168678 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:42.168587    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5a7382-6z7sb" podStartSLOduration=2.168567908 podStartE2EDuration="2.168567908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:14:42.168166692 +0000 UTC m=+2106.437230153" watchObservedRunningTime="2026-11-03 09:14:42.168567908 +0000 UTC m=+2106.437631343"
Nov 03 09:14:43.060251 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:43.060215    4473 generic.go:356] "Generic (PLEG): container finished" podID="447303f1-c173-4810-acfd-51993e615c0f" containerID="4ac96d55c1d9c2f443010b9736bb9d4835c983ff928fb8c1092a034145cc1929" exitCode=0
Nov 03 09:14:43.060717 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:43.060305    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5a7382-6z7sb" event={"ID":"447303f1-c173-4810-acfd-51993e615c0f","Type":"ContainerDied","Data":"4ac96d55c1d9c2f443010b9736bb9d4835c983ff928fb8c1092a034145cc1929"}
Nov 03 09:14:43.777926 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:43.777890    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b64d08a9-h9ph8"]
Nov 03 09:14:43.781809 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:43.781786    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b64d08a9-h9ph8"
Nov 03 09:14:43.840478 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:43.840440    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b64d08a9-h9ph8"]
Nov 03 09:14:43.928152 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:43.928116    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h6v4\" (UniqueName: \"kubernetes.io/projected/a936a741-17a8-45d5-9828-4891c861cc88-kube-api-access-7h6v4\") pod \"maas-api-revoke-sub-b64d08a9-h9ph8\" (UID: \"a936a741-17a8-45d5-9828-4891c861cc88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b64d08a9-h9ph8"
Nov 03 09:14:43.928355 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:43.928212    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a936a741-17a8-45d5-9828-4891c861cc88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b64d08a9-h9ph8\" (UID: \"a936a741-17a8-45d5-9828-4891c861cc88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b64d08a9-h9ph8"
Nov 03 09:14:43.928355 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:43.928243    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a936a741-17a8-45d5-9828-4891c861cc88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b64d08a9-h9ph8\" (UID: \"a936a741-17a8-45d5-9828-4891c861cc88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b64d08a9-h9ph8"
Nov 03 09:14:44.030548 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:44.030421    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a936a741-17a8-45d5-9828-4891c861cc88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b64d08a9-h9ph8\" (UID: \"a936a741-17a8-45d5-9828-4891c861cc88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b64d08a9-h9ph8"
Nov 03 09:14:44.030548 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:44.030516    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a936a741-17a8-45d5-9828-4891c861cc88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b64d08a9-h9ph8\" (UID: \"a936a741-17a8-45d5-9828-4891c861cc88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b64d08a9-h9ph8"
Nov 03 09:14:44.030903 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:44.030872    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h6v4\" (UniqueName: \"kubernetes.io/projected/a936a741-17a8-45d5-9828-4891c861cc88-kube-api-access-7h6v4\") pod \"maas-api-revoke-sub-b64d08a9-h9ph8\" (UID: \"a936a741-17a8-45d5-9828-4891c861cc88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b64d08a9-h9ph8"
Nov 03 09:14:44.031384 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:44.031359    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a936a741-17a8-45d5-9828-4891c861cc88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b64d08a9-h9ph8\" (UID: \"a936a741-17a8-45d5-9828-4891c861cc88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b64d08a9-h9ph8"
Nov 03 09:14:44.033673 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:44.033651    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a936a741-17a8-45d5-9828-4891c861cc88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b64d08a9-h9ph8\" (UID: \"a936a741-17a8-45d5-9828-4891c861cc88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b64d08a9-h9ph8"
Nov 03 09:14:44.091146 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:44.091113    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h6v4\" (UniqueName: \"kubernetes.io/projected/a936a741-17a8-45d5-9828-4891c861cc88-kube-api-access-7h6v4\") pod \"maas-api-revoke-sub-b64d08a9-h9ph8\" (UID: \"a936a741-17a8-45d5-9828-4891c861cc88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b64d08a9-h9ph8"
Nov 03 09:14:44.094790 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:44.094764    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b64d08a9-h9ph8"
Nov 03 09:14:44.237691 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:44.237665    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5a7382-6z7sb"
Nov 03 09:14:44.335191 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:44.335113    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/447303f1-c173-4810-acfd-51993e615c0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/447303f1-c173-4810-acfd-51993e615c0f-maas-api-xks-ca\") pod \"447303f1-c173-4810-acfd-51993e615c0f\" (UID: \"447303f1-c173-4810-acfd-51993e615c0f\") "
Nov 03 09:14:44.335363 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:44.335277    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/447303f1-c173-4810-acfd-51993e615c0f-kube-api-access-fkkw6\" (UniqueName: \"kubernetes.io/projected/447303f1-c173-4810-acfd-51993e615c0f-kube-api-access-fkkw6\") pod \"447303f1-c173-4810-acfd-51993e615c0f\" (UID: \"447303f1-c173-4810-acfd-51993e615c0f\") "
Nov 03 09:14:44.335363 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:44.335330    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/447303f1-c173-4810-acfd-51993e615c0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/447303f1-c173-4810-acfd-51993e615c0f-maas-api-openshift-ca\") pod \"447303f1-c173-4810-acfd-51993e615c0f\" (UID: \"447303f1-c173-4810-acfd-51993e615c0f\") "
Nov 03 09:14:44.335746 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:44.335725    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/447303f1-c173-4810-acfd-51993e615c0f-maas-api-openshift-ca" pod "447303f1-c173-4810-acfd-51993e615c0f" (UID: "447303f1-c173-4810-acfd-51993e615c0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:14:44.337404 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:44.337387    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/447303f1-c173-4810-acfd-51993e615c0f-maas-api-xks-ca" pod "447303f1-c173-4810-acfd-51993e615c0f" (UID: "447303f1-c173-4810-acfd-51993e615c0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:14:44.337472 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:44.337451    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/447303f1-c173-4810-acfd-51993e615c0f-kube-api-access-fkkw6" pod "447303f1-c173-4810-acfd-51993e615c0f" (UID: "447303f1-c173-4810-acfd-51993e615c0f"). InnerVolumeSpecName "kube-api-access-fkkw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:14:44.437022 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:44.436987    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkkw6\" (UniqueName: \"kubernetes.io/projected/447303f1-c173-4810-acfd-51993e615c0f-kube-api-access-fkkw6\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:14:44.437022 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:44.437013    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/447303f1-c173-4810-acfd-51993e615c0f-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:14:44.437022 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:44.437023    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/447303f1-c173-4810-acfd-51993e615c0f-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:14:44.504249 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:44.504223    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b64d08a9-h9ph8"]
Nov 03 09:14:44.506373 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:14:44.506340    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda936a741_17a8_45d5_9828_4891c861cc88.slice/crio-450e24d5e67721d86093bfcf782a628e34f5dcc09505c774fa765c8dd74e5a80 WatchSource:0}: Error finding container 450e24d5e67721d86093bfcf782a628e34f5dcc09505c774fa765c8dd74e5a80: Status 404 returned error can't find the container with id 450e24d5e67721d86093bfcf782a628e34f5dcc09505c774fa765c8dd74e5a80
Nov 03 09:14:45.073750 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:45.073707    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b64d08a9-h9ph8" event={"ID":"a936a741-17a8-45d5-9828-4891c861cc88","Type":"ContainerStarted","Data":"715d2bac9d932f0b33bd5f1f6ce80c5e1c2355917337c2126473cce2d4974b37"}
Nov 03 09:14:45.073980 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:45.073774    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b64d08a9-h9ph8" event={"ID":"a936a741-17a8-45d5-9828-4891c861cc88","Type":"ContainerStarted","Data":"450e24d5e67721d86093bfcf782a628e34f5dcc09505c774fa765c8dd74e5a80"}
Nov 03 09:14:45.075305 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:45.075275    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5a7382-6z7sb" event={"ID":"447303f1-c173-4810-acfd-51993e615c0f","Type":"ContainerDied","Data":"2f4a1226f27bdffc66ec682d2ba875812c031054fc8de4da4beab84b71567166"}
Nov 03 09:14:45.075305 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:45.075299    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd5a7382-6z7sb"
Nov 03 09:14:45.075495 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:45.075326    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f4a1226f27bdffc66ec682d2ba875812c031054fc8de4da4beab84b71567166"
Nov 03 09:14:45.190461 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:45.190404    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b64d08a9-h9ph8" podStartSLOduration=2.190388983 podStartE2EDuration="2.190388983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:14:45.188776335 +0000 UTC m=+2109.457839794" watchObservedRunningTime="2026-11-03 09:14:45.190388983 +0000 UTC m=+2109.459452417"
Nov 03 09:14:46.082424 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:46.082389    4473 generic.go:356] "Generic (PLEG): container finished" podID="a936a741-17a8-45d5-9828-4891c861cc88" containerID="715d2bac9d932f0b33bd5f1f6ce80c5e1c2355917337c2126473cce2d4974b37" exitCode=0
Nov 03 09:14:46.082601 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:46.082480    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b64d08a9-h9ph8" event={"ID":"a936a741-17a8-45d5-9828-4891c861cc88","Type":"ContainerDied","Data":"715d2bac9d932f0b33bd5f1f6ce80c5e1c2355917337c2126473cce2d4974b37"}
Nov 03 09:14:46.936771 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:14:46.936678    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:14:47.242219 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:47.242197    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b64d08a9-h9ph8"
Nov 03 09:14:47.268714 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:47.268672    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a936a741-17a8-45d5-9828-4891c861cc88-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a936a741-17a8-45d5-9828-4891c861cc88-maas-api-xks-ca\") pod \"a936a741-17a8-45d5-9828-4891c861cc88\" (UID: \"a936a741-17a8-45d5-9828-4891c861cc88\") "
Nov 03 09:14:47.268857 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:47.268779    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a936a741-17a8-45d5-9828-4891c861cc88-kube-api-access-7h6v4\" (UniqueName: \"kubernetes.io/projected/a936a741-17a8-45d5-9828-4891c861cc88-kube-api-access-7h6v4\") pod \"a936a741-17a8-45d5-9828-4891c861cc88\" (UID: \"a936a741-17a8-45d5-9828-4891c861cc88\") "
Nov 03 09:14:47.268857 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:47.268797    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a936a741-17a8-45d5-9828-4891c861cc88-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a936a741-17a8-45d5-9828-4891c861cc88-maas-api-openshift-ca\") pod \"a936a741-17a8-45d5-9828-4891c861cc88\" (UID: \"a936a741-17a8-45d5-9828-4891c861cc88\") "
Nov 03 09:14:47.269279 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:47.269260    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a936a741-17a8-45d5-9828-4891c861cc88-maas-api-openshift-ca" pod "a936a741-17a8-45d5-9828-4891c861cc88" (UID: "a936a741-17a8-45d5-9828-4891c861cc88"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:14:47.270887 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:47.270865    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a936a741-17a8-45d5-9828-4891c861cc88-maas-api-xks-ca" pod "a936a741-17a8-45d5-9828-4891c861cc88" (UID: "a936a741-17a8-45d5-9828-4891c861cc88"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:14:47.271062 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:47.271041    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a936a741-17a8-45d5-9828-4891c861cc88-kube-api-access-7h6v4" pod "a936a741-17a8-45d5-9828-4891c861cc88" (UID: "a936a741-17a8-45d5-9828-4891c861cc88"). InnerVolumeSpecName "kube-api-access-7h6v4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:14:47.370227 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:47.370194    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a936a741-17a8-45d5-9828-4891c861cc88-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:14:47.370227 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:47.370222    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h6v4\" (UniqueName: \"kubernetes.io/projected/a936a741-17a8-45d5-9828-4891c861cc88-kube-api-access-7h6v4\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:14:47.370227 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:47.370232    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a936a741-17a8-45d5-9828-4891c861cc88-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:14:48.094940 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:48.094912    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b64d08a9-h9ph8"
Nov 03 09:14:48.095353 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:48.094931    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b64d08a9-h9ph8" event={"ID":"a936a741-17a8-45d5-9828-4891c861cc88","Type":"ContainerDied","Data":"450e24d5e67721d86093bfcf782a628e34f5dcc09505c774fa765c8dd74e5a80"}
Nov 03 09:14:48.095353 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:48.095012    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="450e24d5e67721d86093bfcf782a628e34f5dcc09505c774fa765c8dd74e5a80"
Nov 03 09:14:52.819153 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:52.819120    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:14:56.581434 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:56.581394    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5de7da4-r4976"]
Nov 03 09:14:56.581904 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:56.581890    4473 cpu_manager.go:414] "containerMap: removing container" podUID="a936a741-17a8-45d5-9828-4891c861cc88" containerName="revoke-keys"
Nov 03 09:14:56.581962 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:56.581904    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a936a741-17a8-45d5-9828-4891c861cc88" containerName="revoke-keys"
Nov 03 09:14:56.581962 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:56.581920    4473 cpu_manager.go:414] "containerMap: removing container" podUID="447303f1-c173-4810-acfd-51993e615c0f" containerName="revoke-keys"
Nov 03 09:14:56.581962 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:56.581925    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="447303f1-c173-4810-acfd-51993e615c0f" containerName="revoke-keys"
Nov 03 09:14:56.582058 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:56.582037    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="a936a741-17a8-45d5-9828-4891c861cc88" containerName="revoke-keys"
Nov 03 09:14:56.582058 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:56.582048    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="447303f1-c173-4810-acfd-51993e615c0f" containerName="revoke-keys"
Nov 03 09:14:56.586409 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:56.586387    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5de7da4-r4976"
Nov 03 09:14:56.642702 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:56.642675    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5de7da4-r4976"]
Nov 03 09:14:56.689469 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:56.689436    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:14:56.689469 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:56.689458    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:14:56.689657 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:56.689541    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:14:56.691680 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:56.691661    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:14:56.726736 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:56.726707    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5de7da4-r4976\" (UID: \"31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5de7da4-r4976"
Nov 03 09:14:56.726966 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:56.726909    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvllt\" (UniqueName: \"kubernetes.io/projected/31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2-kube-api-access-zvllt\") pod \"maas-api-revoke-sub-a5de7da4-r4976\" (UID: \"31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5de7da4-r4976"
Nov 03 09:14:56.727116 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:56.727021    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5de7da4-r4976\" (UID: \"31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5de7da4-r4976"
Nov 03 09:14:56.827865 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:56.827835    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvllt\" (UniqueName: \"kubernetes.io/projected/31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2-kube-api-access-zvllt\") pod \"maas-api-revoke-sub-a5de7da4-r4976\" (UID: \"31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5de7da4-r4976"
Nov 03 09:14:56.828094 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:56.827934    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5de7da4-r4976\" (UID: \"31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5de7da4-r4976"
Nov 03 09:14:56.828094 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:56.828064    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5de7da4-r4976\" (UID: \"31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5de7da4-r4976"
Nov 03 09:14:56.828770 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:56.828742    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5de7da4-r4976\" (UID: \"31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5de7da4-r4976"
Nov 03 09:14:56.830939 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:56.830915    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5de7da4-r4976\" (UID: \"31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5de7da4-r4976"
Nov 03 09:14:56.887870 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:56.887845    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvllt\" (UniqueName: \"kubernetes.io/projected/31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2-kube-api-access-zvllt\") pod \"maas-api-revoke-sub-a5de7da4-r4976\" (UID: \"31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5de7da4-r4976"
Nov 03 09:14:56.899708 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:56.899677    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5de7da4-r4976"
Nov 03 09:14:57.275602 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:57.275576    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5de7da4-r4976"]
Nov 03 09:14:57.276471 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:14:57.276446    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31d9edb0_a5e7_4f7d_9fac_3e1c1f62c5c2.slice/crio-a5e143e8df6a3a80dd440eaefb18f59496843110bf09d6a4c8e10d3743de8aa6 WatchSource:0}: Error finding container a5e143e8df6a3a80dd440eaefb18f59496843110bf09d6a4c8e10d3743de8aa6: Status 404 returned error can't find the container with id a5e143e8df6a3a80dd440eaefb18f59496843110bf09d6a4c8e10d3743de8aa6
Nov 03 09:14:58.178112 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:58.178072    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5de7da4-r4976" event={"ID":"31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2","Type":"ContainerStarted","Data":"6a24801ebad9e468705c1d09818464d6efed7b4863b37299c524f3ed50b62cbf"}
Nov 03 09:14:58.178586 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:58.178135    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5de7da4-r4976" event={"ID":"31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2","Type":"ContainerStarted","Data":"a5e143e8df6a3a80dd440eaefb18f59496843110bf09d6a4c8e10d3743de8aa6"}
Nov 03 09:14:58.296044 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:58.295977    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5de7da4-r4976" podStartSLOduration=2.295939688 podStartE2EDuration="2.295939688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:14:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:14:58.294733306 +0000 UTC m=+2122.563796753" watchObservedRunningTime="2026-11-03 09:14:58.295939688 +0000 UTC m=+2122.565003119"
Nov 03 09:14:59.186392 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:59.186358    4473 generic.go:356] "Generic (PLEG): container finished" podID="31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2" containerID="6a24801ebad9e468705c1d09818464d6efed7b4863b37299c524f3ed50b62cbf" exitCode=0
Nov 03 09:14:59.186839 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:14:59.186440    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5de7da4-r4976" event={"ID":"31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2","Type":"ContainerDied","Data":"6a24801ebad9e468705c1d09818464d6efed7b4863b37299c524f3ed50b62cbf"}
Nov 03 09:15:00.166106 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.166076    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894955-777bf"]
Nov 03 09:15:00.170136 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.170114    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04934e-29894955-2z4nq"]
Nov 03 09:15:00.170301 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.170241    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894955-777bf"
Nov 03 09:15:00.174183 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.174163    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894955-7v5xg"]
Nov 03 09:15:00.174311 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.174270    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04934e-29894955-2z4nq"
Nov 03 09:15:00.178401 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.178347    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894955-7v5xg"
Nov 03 09:15:00.216549 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.216513    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894955-58wlg"]
Nov 03 09:15:00.220489 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.220467    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894955-58wlg"
Nov 03 09:15:00.220873 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.220853    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894955-z25mc"]
Nov 03 09:15:00.224487 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.224464    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-133b31-29894955-wtst5"]
Nov 03 09:15:00.224600 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.224561    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894955-z25mc"
Nov 03 09:15:00.230466 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.229342    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-121ee6-29894955-6nz68"]
Nov 03 09:15:00.230466 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.229475    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-133b31-29894955-wtst5"
Nov 03 09:15:00.233220 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.233203    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-121ee6-29894955-6nz68"
Nov 03 09:15:00.267435 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.267407    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894955-777bf"]
Nov 03 09:15:00.269719 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.269690    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c5x5\" (UniqueName: \"kubernetes.io/projected/750847c9-fd9f-4d39-8705-ccea1fd3fc17-kube-api-access-4c5x5\") pod \"maas-api-key-cleanup-e2e-worker-w2-77207e-29894955-7v5xg\" (UID: \"750847c9-fd9f-4d39-8705-ccea1fd3fc17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894955-7v5xg"
Nov 03 09:15:00.269877 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.269846    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfrw8\" (UniqueName: \"kubernetes.io/projected/20ec7053-908f-46d0-8b5e-f3fdb57d433e-kube-api-access-tfrw8\") pod \"maas-api-key-cleanup-e2e-worker-w1-26e861-29894955-777bf\" (UID: \"20ec7053-908f-46d0-8b5e-f3fdb57d433e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894955-777bf"
Nov 03 09:15:00.270014 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.269961    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb6573f4-bd1c-4383-bb08-7fdae8f679d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-04934e-29894955-2z4nq\" (UID: \"cb6573f4-bd1c-4383-bb08-7fdae8f679d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04934e-29894955-2z4nq"
Nov 03 09:15:00.270169 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.270151    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20ec7053-908f-46d0-8b5e-f3fdb57d433e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-26e861-29894955-777bf\" (UID: \"20ec7053-908f-46d0-8b5e-f3fdb57d433e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894955-777bf"
Nov 03 09:15:00.270304 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.270272    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2rc2\" (UniqueName: \"kubernetes.io/projected/cb6573f4-bd1c-4383-bb08-7fdae8f679d2-kube-api-access-p2rc2\") pod \"maas-api-key-cleanup-e2e-ipp-a-04934e-29894955-2z4nq\" (UID: \"cb6573f4-bd1c-4383-bb08-7fdae8f679d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04934e-29894955-2z4nq"
Nov 03 09:15:00.270394 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.270310    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/750847c9-fd9f-4d39-8705-ccea1fd3fc17-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-77207e-29894955-7v5xg\" (UID: \"750847c9-fd9f-4d39-8705-ccea1fd3fc17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894955-7v5xg"
Nov 03 09:15:00.270457 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.270447    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04934e-29894955-2z4nq"]
Nov 03 09:15:00.273182 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.273159    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894955-7v5xg"]
Nov 03 09:15:00.349007 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.328476    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894955-58wlg"]
Nov 03 09:15:00.389059 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.374994    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61f3f0ff-1cbf-4e08-8d44-f44952e3b01c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-18f9da-29894955-58wlg\" (UID: \"61f3f0ff-1cbf-4e08-8d44-f44952e3b01c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894955-58wlg"
Nov 03 09:15:00.389059 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.375058    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00db03c4-9076-4e59-a799-a260f34a1e7d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-121ee6-29894955-6nz68\" (UID: \"00db03c4-9076-4e59-a799-a260f34a1e7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-121ee6-29894955-6nz68"
Nov 03 09:15:00.389059 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.375122    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2rc2\" (UniqueName: \"kubernetes.io/projected/cb6573f4-bd1c-4383-bb08-7fdae8f679d2-kube-api-access-p2rc2\") pod \"maas-api-key-cleanup-e2e-ipp-a-04934e-29894955-2z4nq\" (UID: \"cb6573f4-bd1c-4383-bb08-7fdae8f679d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04934e-29894955-2z4nq"
Nov 03 09:15:00.389059 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.375149    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/750847c9-fd9f-4d39-8705-ccea1fd3fc17-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-77207e-29894955-7v5xg\" (UID: \"750847c9-fd9f-4d39-8705-ccea1fd3fc17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894955-7v5xg"
Nov 03 09:15:00.389059 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.375196    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c5x5\" (UniqueName: \"kubernetes.io/projected/750847c9-fd9f-4d39-8705-ccea1fd3fc17-kube-api-access-4c5x5\") pod \"maas-api-key-cleanup-e2e-worker-w2-77207e-29894955-7v5xg\" (UID: \"750847c9-fd9f-4d39-8705-ccea1fd3fc17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894955-7v5xg"
Nov 03 09:15:00.389059 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.375249    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf19596b-0d54-4f5f-a6d9-433b4c130cbc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-133b31-29894955-wtst5\" (UID: \"bf19596b-0d54-4f5f-a6d9-433b4c130cbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-133b31-29894955-wtst5"
Nov 03 09:15:00.389059 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.375274    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf96s\" (UniqueName: \"kubernetes.io/projected/00db03c4-9076-4e59-a799-a260f34a1e7d-kube-api-access-cf96s\") pod \"maas-api-key-cleanup-e2e-ipp-b-121ee6-29894955-6nz68\" (UID: \"00db03c4-9076-4e59-a799-a260f34a1e7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-121ee6-29894955-6nz68"
Nov 03 09:15:00.389059 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.375343    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfrw8\" (UniqueName: \"kubernetes.io/projected/20ec7053-908f-46d0-8b5e-f3fdb57d433e-kube-api-access-tfrw8\") pod \"maas-api-key-cleanup-e2e-worker-w1-26e861-29894955-777bf\" (UID: \"20ec7053-908f-46d0-8b5e-f3fdb57d433e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894955-777bf"
Nov 03 09:15:00.389059 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.375377    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb6573f4-bd1c-4383-bb08-7fdae8f679d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-04934e-29894955-2z4nq\" (UID: \"cb6573f4-bd1c-4383-bb08-7fdae8f679d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04934e-29894955-2z4nq"
Nov 03 09:15:00.389059 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.375429    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxdsx\" (UniqueName: \"kubernetes.io/projected/bf19596b-0d54-4f5f-a6d9-433b4c130cbc-kube-api-access-kxdsx\") pod \"maas-api-key-cleanup-e2e-praxis-a-133b31-29894955-wtst5\" (UID: \"bf19596b-0d54-4f5f-a6d9-433b4c130cbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-133b31-29894955-wtst5"
Nov 03 09:15:00.389059 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.375470    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k8z2\" (UniqueName: \"kubernetes.io/projected/4cca539f-d6ea-4402-ac67-28886b17c397-kube-api-access-8k8z2\") pod \"maas-api-key-cleanup-e2e-worker-w0-910d50-29894955-z25mc\" (UID: \"4cca539f-d6ea-4402-ac67-28886b17c397\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894955-z25mc"
Nov 03 09:15:00.389059 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.375516    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctpl2\" (UniqueName: \"kubernetes.io/projected/61f3f0ff-1cbf-4e08-8d44-f44952e3b01c-kube-api-access-ctpl2\") pod \"maas-api-key-cleanup-e2e-worker-w4-18f9da-29894955-58wlg\" (UID: \"61f3f0ff-1cbf-4e08-8d44-f44952e3b01c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894955-58wlg"
Nov 03 09:15:00.389059 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.375626    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cca539f-d6ea-4402-ac67-28886b17c397-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-910d50-29894955-z25mc\" (UID: \"4cca539f-d6ea-4402-ac67-28886b17c397\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894955-z25mc"
Nov 03 09:15:00.389059 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.375671    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20ec7053-908f-46d0-8b5e-f3fdb57d433e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-26e861-29894955-777bf\" (UID: \"20ec7053-908f-46d0-8b5e-f3fdb57d433e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894955-777bf"
Nov 03 09:15:00.409971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.390986    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-121ee6-29894955-6nz68"]
Nov 03 09:15:00.409971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.391064    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-133b31-29894955-wtst5"]
Nov 03 09:15:00.409971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.392180    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/750847c9-fd9f-4d39-8705-ccea1fd3fc17-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-77207e-29894955-7v5xg\" (UID: \"750847c9-fd9f-4d39-8705-ccea1fd3fc17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894955-7v5xg"
Nov 03 09:15:00.433707 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.392925    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb6573f4-bd1c-4383-bb08-7fdae8f679d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-04934e-29894955-2z4nq\" (UID: \"cb6573f4-bd1c-4383-bb08-7fdae8f679d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04934e-29894955-2z4nq"
Nov 03 09:15:00.433707 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.420494    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894955-z25mc"]
Nov 03 09:15:00.433707 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.430730    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20ec7053-908f-46d0-8b5e-f3fdb57d433e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-26e861-29894955-777bf\" (UID: \"20ec7053-908f-46d0-8b5e-f3fdb57d433e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894955-777bf"
Nov 03 09:15:00.455457 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.455419    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2rc2\" (UniqueName: \"kubernetes.io/projected/cb6573f4-bd1c-4383-bb08-7fdae8f679d2-kube-api-access-p2rc2\") pod \"maas-api-key-cleanup-e2e-ipp-a-04934e-29894955-2z4nq\" (UID: \"cb6573f4-bd1c-4383-bb08-7fdae8f679d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04934e-29894955-2z4nq"
Nov 03 09:15:00.456285 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.456256    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c5x5\" (UniqueName: \"kubernetes.io/projected/750847c9-fd9f-4d39-8705-ccea1fd3fc17-kube-api-access-4c5x5\") pod \"maas-api-key-cleanup-e2e-worker-w2-77207e-29894955-7v5xg\" (UID: \"750847c9-fd9f-4d39-8705-ccea1fd3fc17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894955-7v5xg"
Nov 03 09:15:00.456846 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.456821    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfrw8\" (UniqueName: \"kubernetes.io/projected/20ec7053-908f-46d0-8b5e-f3fdb57d433e-kube-api-access-tfrw8\") pod \"maas-api-key-cleanup-e2e-worker-w1-26e861-29894955-777bf\" (UID: \"20ec7053-908f-46d0-8b5e-f3fdb57d433e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894955-777bf"
Nov 03 09:15:00.477427 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.476992    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxdsx\" (UniqueName: \"kubernetes.io/projected/bf19596b-0d54-4f5f-a6d9-433b4c130cbc-kube-api-access-kxdsx\") pod \"maas-api-key-cleanup-e2e-praxis-a-133b31-29894955-wtst5\" (UID: \"bf19596b-0d54-4f5f-a6d9-433b4c130cbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-133b31-29894955-wtst5"
Nov 03 09:15:00.477427 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.477050    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k8z2\" (UniqueName: \"kubernetes.io/projected/4cca539f-d6ea-4402-ac67-28886b17c397-kube-api-access-8k8z2\") pod \"maas-api-key-cleanup-e2e-worker-w0-910d50-29894955-z25mc\" (UID: \"4cca539f-d6ea-4402-ac67-28886b17c397\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894955-z25mc"
Nov 03 09:15:00.477427 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.477080    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctpl2\" (UniqueName: \"kubernetes.io/projected/61f3f0ff-1cbf-4e08-8d44-f44952e3b01c-kube-api-access-ctpl2\") pod \"maas-api-key-cleanup-e2e-worker-w4-18f9da-29894955-58wlg\" (UID: \"61f3f0ff-1cbf-4e08-8d44-f44952e3b01c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894955-58wlg"
Nov 03 09:15:00.477427 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.477145    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cca539f-d6ea-4402-ac67-28886b17c397-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-910d50-29894955-z25mc\" (UID: \"4cca539f-d6ea-4402-ac67-28886b17c397\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894955-z25mc"
Nov 03 09:15:00.477427 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.477179    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61f3f0ff-1cbf-4e08-8d44-f44952e3b01c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-18f9da-29894955-58wlg\" (UID: \"61f3f0ff-1cbf-4e08-8d44-f44952e3b01c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894955-58wlg"
Nov 03 09:15:00.477427 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.477208    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00db03c4-9076-4e59-a799-a260f34a1e7d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-121ee6-29894955-6nz68\" (UID: \"00db03c4-9076-4e59-a799-a260f34a1e7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-121ee6-29894955-6nz68"
Nov 03 09:15:00.477427 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.477293    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf19596b-0d54-4f5f-a6d9-433b4c130cbc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-133b31-29894955-wtst5\" (UID: \"bf19596b-0d54-4f5f-a6d9-433b4c130cbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-133b31-29894955-wtst5"
Nov 03 09:15:00.477427 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.477308    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf96s\" (UniqueName: \"kubernetes.io/projected/00db03c4-9076-4e59-a799-a260f34a1e7d-kube-api-access-cf96s\") pod \"maas-api-key-cleanup-e2e-ipp-b-121ee6-29894955-6nz68\" (UID: \"00db03c4-9076-4e59-a799-a260f34a1e7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-121ee6-29894955-6nz68"
Nov 03 09:15:00.478793 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.478761    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00db03c4-9076-4e59-a799-a260f34a1e7d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-121ee6-29894955-6nz68\" (UID: \"00db03c4-9076-4e59-a799-a260f34a1e7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-121ee6-29894955-6nz68"
Nov 03 09:15:00.478926 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.478761    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cca539f-d6ea-4402-ac67-28886b17c397-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-910d50-29894955-z25mc\" (UID: \"4cca539f-d6ea-4402-ac67-28886b17c397\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894955-z25mc"
Nov 03 09:15:00.478926 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.478769    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf19596b-0d54-4f5f-a6d9-433b4c130cbc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-133b31-29894955-wtst5\" (UID: \"bf19596b-0d54-4f5f-a6d9-433b4c130cbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-133b31-29894955-wtst5"
Nov 03 09:15:00.479319 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.479299    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61f3f0ff-1cbf-4e08-8d44-f44952e3b01c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-18f9da-29894955-58wlg\" (UID: \"61f3f0ff-1cbf-4e08-8d44-f44952e3b01c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894955-58wlg"
Nov 03 09:15:00.485564 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.485542    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894955-777bf"
Nov 03 09:15:00.486744 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.486720    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5de7da4-r4976"
Nov 03 09:15:00.498450 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.498367    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04934e-29894955-2z4nq"
Nov 03 09:15:00.507766 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.507690    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894955-7v5xg"
Nov 03 09:15:00.554352 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.551683    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctpl2\" (UniqueName: \"kubernetes.io/projected/61f3f0ff-1cbf-4e08-8d44-f44952e3b01c-kube-api-access-ctpl2\") pod \"maas-api-key-cleanup-e2e-worker-w4-18f9da-29894955-58wlg\" (UID: \"61f3f0ff-1cbf-4e08-8d44-f44952e3b01c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894955-58wlg"
Nov 03 09:15:00.556176 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.555241    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxdsx\" (UniqueName: \"kubernetes.io/projected/bf19596b-0d54-4f5f-a6d9-433b4c130cbc-kube-api-access-kxdsx\") pod \"maas-api-key-cleanup-e2e-praxis-a-133b31-29894955-wtst5\" (UID: \"bf19596b-0d54-4f5f-a6d9-433b4c130cbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-133b31-29894955-wtst5"
Nov 03 09:15:00.559639 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.559121    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-133b31-29894955-wtst5"
Nov 03 09:15:00.577376 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.562241    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894955-58wlg"
Nov 03 09:15:00.577376 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.570231    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf96s\" (UniqueName: \"kubernetes.io/projected/00db03c4-9076-4e59-a799-a260f34a1e7d-kube-api-access-cf96s\") pod \"maas-api-key-cleanup-e2e-ipp-b-121ee6-29894955-6nz68\" (UID: \"00db03c4-9076-4e59-a799-a260f34a1e7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-121ee6-29894955-6nz68"
Nov 03 09:15:00.577376 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.571796    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-121ee6-29894955-6nz68"
Nov 03 09:15:00.577376 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.573096    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k8z2\" (UniqueName: \"kubernetes.io/projected/4cca539f-d6ea-4402-ac67-28886b17c397-kube-api-access-8k8z2\") pod \"maas-api-key-cleanup-e2e-worker-w0-910d50-29894955-z25mc\" (UID: \"4cca539f-d6ea-4402-ac67-28886b17c397\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894955-z25mc"
Nov 03 09:15:00.580297 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.580269    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2-kube-api-access-zvllt\" (UniqueName: \"kubernetes.io/projected/31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2-kube-api-access-zvllt\") pod \"31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2\" (UID: \"31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2\") "
Nov 03 09:15:00.580458 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.580446    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2-maas-api-openshift-ca\") pod \"31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2\" (UID: \"31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2\") "
Nov 03 09:15:00.580506 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.580485    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2-maas-api-xks-ca\") pod \"31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2\" (UID: \"31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2\") "
Nov 03 09:15:00.582094 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.582064    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2-maas-api-openshift-ca" pod "31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2" (UID: "31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:15:00.583073 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.583043    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2-maas-api-xks-ca" pod "31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2" (UID: "31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:15:00.583630 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.583609    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2-kube-api-access-zvllt" pod "31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2" (UID: "31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2"). InnerVolumeSpecName "kube-api-access-zvllt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:15:00.647583 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.647506    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-0eecfd-76cbfb4455-4ztkx"]
Nov 03 09:15:00.649323 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.649300    4473 cpu_manager.go:414] "containerMap: removing container" podUID="31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2" containerName="revoke-keys"
Nov 03 09:15:00.649487 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.649476    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2" containerName="revoke-keys"
Nov 03 09:15:00.649683 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.649669    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2" containerName="revoke-keys"
Nov 03 09:15:00.674489 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.670099    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0eecfd-76cbfb4455-4ztkx"
Nov 03 09:15:00.682310 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.682232    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:00.682310 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.682266    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:00.682310 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.682285    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvllt\" (UniqueName: \"kubernetes.io/projected/31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2-kube-api-access-zvllt\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:00.742562 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.730277    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-0eecfd-dockercfg-7vlhk\""
Nov 03 09:15:00.742562 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.730373    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-0eecfd\""
Nov 03 09:15:00.797248 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.796775    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjvrv\" (UniqueName: \"kubernetes.io/projected/6cb96f19-e812-48af-9995-3fdc60b20e15-kube-api-access-bjvrv\") pod \"payload-pre-processing-e2e-praxis-b-0eecfd-76cbfb4455-4ztkx\" (UID: \"6cb96f19-e812-48af-9995-3fdc60b20e15\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0eecfd-76cbfb4455-4ztkx"
Nov 03 09:15:00.797248 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.797042    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb96f19-e812-48af-9995-3fdc60b20e15-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-0eecfd-76cbfb4455-4ztkx\" (UID: \"6cb96f19-e812-48af-9995-3fdc60b20e15\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0eecfd-76cbfb4455-4ztkx"
Nov 03 09:15:00.869968 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.864923    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894955-z25mc"
Nov 03 09:15:00.876090 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.867251    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-0eecfd-76cbfb4455-4ztkx"]
Nov 03 09:15:00.898036 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.898011    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjvrv\" (UniqueName: \"kubernetes.io/projected/6cb96f19-e812-48af-9995-3fdc60b20e15-kube-api-access-bjvrv\") pod \"payload-pre-processing-e2e-praxis-b-0eecfd-76cbfb4455-4ztkx\" (UID: \"6cb96f19-e812-48af-9995-3fdc60b20e15\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0eecfd-76cbfb4455-4ztkx"
Nov 03 09:15:00.898135 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.898066    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb96f19-e812-48af-9995-3fdc60b20e15-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-0eecfd-76cbfb4455-4ztkx\" (UID: \"6cb96f19-e812-48af-9995-3fdc60b20e15\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0eecfd-76cbfb4455-4ztkx"
Nov 03 09:15:00.898714 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.898694    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb96f19-e812-48af-9995-3fdc60b20e15-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-0eecfd-76cbfb4455-4ztkx\" (UID: \"6cb96f19-e812-48af-9995-3fdc60b20e15\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0eecfd-76cbfb4455-4ztkx"
Nov 03 09:15:00.960985 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:00.960696    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjvrv\" (UniqueName: \"kubernetes.io/projected/6cb96f19-e812-48af-9995-3fdc60b20e15-kube-api-access-bjvrv\") pod \"payload-pre-processing-e2e-praxis-b-0eecfd-76cbfb4455-4ztkx\" (UID: \"6cb96f19-e812-48af-9995-3fdc60b20e15\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0eecfd-76cbfb4455-4ztkx"
Nov 03 09:15:01.043818 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:01.043783    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0eecfd-76cbfb4455-4ztkx"
Nov 03 09:15:01.194962 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:01.194921    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894955-7v5xg"]
Nov 03 09:15:01.196160 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:15:01.196138    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod750847c9_fd9f_4d39_8705_ccea1fd3fc17.slice/crio-8ab98ce00b9a9b22f271e850dee6ec5222c515c1ce414bb85ffb93b3acef8af8 WatchSource:0}: Error finding container 8ab98ce00b9a9b22f271e850dee6ec5222c515c1ce414bb85ffb93b3acef8af8: Status 404 returned error can't find the container with id 8ab98ce00b9a9b22f271e850dee6ec5222c515c1ce414bb85ffb93b3acef8af8
Nov 03 09:15:01.200574 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:01.200491    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5de7da4-r4976"
Nov 03 09:15:01.200574 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:01.200531    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5de7da4-r4976" event={"ID":"31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2","Type":"ContainerDied","Data":"a5e143e8df6a3a80dd440eaefb18f59496843110bf09d6a4c8e10d3743de8aa6"}
Nov 03 09:15:01.200574 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:01.200571    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5e143e8df6a3a80dd440eaefb18f59496843110bf09d6a4c8e10d3743de8aa6"
Nov 03 09:15:01.226142 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:01.226024    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04934e-29894955-2z4nq"]
Nov 03 09:15:01.228504 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:15:01.228480    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb6573f4_bd1c_4383_bb08_7fdae8f679d2.slice/crio-6fa99d368ee46c3a362849f05a120d2c17ae6939abfc80695dc3bfce94f0ddaa WatchSource:0}: Error finding container 6fa99d368ee46c3a362849f05a120d2c17ae6939abfc80695dc3bfce94f0ddaa: Status 404 returned error can't find the container with id 6fa99d368ee46c3a362849f05a120d2c17ae6939abfc80695dc3bfce94f0ddaa
Nov 03 09:15:01.229569 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:01.229546    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894955-777bf"]
Nov 03 09:15:01.229910 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:15:01.229889    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20ec7053_908f_46d0_8b5e_f3fdb57d433e.slice/crio-52d8cc0e7ab436824c54a220394ae05bc23c9b9a1945d7923f063bdae30c57a1 WatchSource:0}: Error finding container 52d8cc0e7ab436824c54a220394ae05bc23c9b9a1945d7923f063bdae30c57a1: Status 404 returned error can't find the container with id 52d8cc0e7ab436824c54a220394ae05bc23c9b9a1945d7923f063bdae30c57a1
Nov 03 09:15:01.245742 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:01.245275    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-121ee6-29894955-6nz68"]
Nov 03 09:15:01.249505 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:01.249478    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894955-58wlg"]
Nov 03 09:15:01.250161 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:15:01.250059    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00db03c4_9076_4e59_a799_a260f34a1e7d.slice/crio-8ca8be60c5639ee27c7eeeb98ac760683b50a94319de78e73c422094c58ecd04 WatchSource:0}: Error finding container 8ca8be60c5639ee27c7eeeb98ac760683b50a94319de78e73c422094c58ecd04: Status 404 returned error can't find the container with id 8ca8be60c5639ee27c7eeeb98ac760683b50a94319de78e73c422094c58ecd04
Nov 03 09:15:01.250365 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:15:01.250341    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f3f0ff_1cbf_4e08_8d44_f44952e3b01c.slice/crio-128f0c019279a21961e7fd4ded531a7477ac3ec123be16f28641f0e3235e2243 WatchSource:0}: Error finding container 128f0c019279a21961e7fd4ded531a7477ac3ec123be16f28641f0e3235e2243: Status 404 returned error can't find the container with id 128f0c019279a21961e7fd4ded531a7477ac3ec123be16f28641f0e3235e2243
Nov 03 09:15:01.283342 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:01.282621    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-133b31-29894955-wtst5"]
Nov 03 09:15:01.286863 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:01.286839    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894955-z25mc"]
Nov 03 09:15:01.288385 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:15:01.288356    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf19596b_0d54_4f5f_a6d9_433b4c130cbc.slice/crio-912d37f877299c2b62af69417e05f1bb0409807a263066e948b8fcf432e2e7bd WatchSource:0}: Error finding container 912d37f877299c2b62af69417e05f1bb0409807a263066e948b8fcf432e2e7bd: Status 404 returned error can't find the container with id 912d37f877299c2b62af69417e05f1bb0409807a263066e948b8fcf432e2e7bd
Nov 03 09:15:01.288596 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:15:01.288579    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cca539f_d6ea_4402_ac67_28886b17c397.slice/crio-d7a68b843cb633d5125e791691df66666b8d2d9c205d67288e682275643817b6 WatchSource:0}: Error finding container d7a68b843cb633d5125e791691df66666b8d2d9c205d67288e682275643817b6: Status 404 returned error can't find the container with id d7a68b843cb633d5125e791691df66666b8d2d9c205d67288e682275643817b6
Nov 03 09:15:01.425201 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:01.425174    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-0eecfd-76cbfb4455-4ztkx"]
Nov 03 09:15:01.521460 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:15:01.521365    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb96f19_e812_48af_9995_3fdc60b20e15.slice/crio-04f13ead6ec475f5bca6973ea500da34802048a4dd7af0100fd56d64f9c6d545 WatchSource:0}: Error finding container 04f13ead6ec475f5bca6973ea500da34802048a4dd7af0100fd56d64f9c6d545: Status 404 returned error can't find the container with id 04f13ead6ec475f5bca6973ea500da34802048a4dd7af0100fd56d64f9c6d545
Nov 03 09:15:02.208258 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:02.208224    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894955-58wlg" event={"ID":"61f3f0ff-1cbf-4e08-8d44-f44952e3b01c","Type":"ContainerStarted","Data":"13b30ed6d5e49d84baeb66a44985c26256e2cff7e495fe4e62f20f3977d2416d"}
Nov 03 09:15:02.208461 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:02.208281    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894955-58wlg" event={"ID":"61f3f0ff-1cbf-4e08-8d44-f44952e3b01c","Type":"ContainerStarted","Data":"128f0c019279a21961e7fd4ded531a7477ac3ec123be16f28641f0e3235e2243"}
Nov 03 09:15:02.209940 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:02.209913    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0eecfd-76cbfb4455-4ztkx" event={"ID":"6cb96f19-e812-48af-9995-3fdc60b20e15","Type":"ContainerStarted","Data":"5a45530a76d4c05527d7761c70e208faed5f9eb44898a9ebe2cbb21fba29361d"}
Nov 03 09:15:02.210072 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:02.209998    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0eecfd-76cbfb4455-4ztkx"
Nov 03 09:15:02.210072 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:02.210026    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0eecfd-76cbfb4455-4ztkx" event={"ID":"6cb96f19-e812-48af-9995-3fdc60b20e15","Type":"ContainerStarted","Data":"04f13ead6ec475f5bca6973ea500da34802048a4dd7af0100fd56d64f9c6d545"}
Nov 03 09:15:02.211614 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:02.211592    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-133b31-29894955-wtst5" event={"ID":"bf19596b-0d54-4f5f-a6d9-433b4c130cbc","Type":"ContainerStarted","Data":"b98125884ecb604c97dac1319f68641cdb1303cb15640f96096ec9e54a1e2d42"}
Nov 03 09:15:02.211734 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:02.211634    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-133b31-29894955-wtst5" event={"ID":"bf19596b-0d54-4f5f-a6d9-433b4c130cbc","Type":"ContainerStarted","Data":"912d37f877299c2b62af69417e05f1bb0409807a263066e948b8fcf432e2e7bd"}
Nov 03 09:15:02.213223 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:02.213192    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04934e-29894955-2z4nq" event={"ID":"cb6573f4-bd1c-4383-bb08-7fdae8f679d2","Type":"ContainerStarted","Data":"fd3d356cac2a425d55f89fa691bef8f592511cfcc5fca1d708f3e76476482680"}
Nov 03 09:15:02.213318 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:02.213233    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04934e-29894955-2z4nq" event={"ID":"cb6573f4-bd1c-4383-bb08-7fdae8f679d2","Type":"ContainerStarted","Data":"6fa99d368ee46c3a362849f05a120d2c17ae6939abfc80695dc3bfce94f0ddaa"}
Nov 03 09:15:02.214675 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:02.214649    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894955-777bf" event={"ID":"20ec7053-908f-46d0-8b5e-f3fdb57d433e","Type":"ContainerStarted","Data":"223b897638e0503d3beabfaa14eca6c0861a88491b39fee5b10cbdf00cf8b88d"}
Nov 03 09:15:02.214776 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:02.214694    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894955-777bf" event={"ID":"20ec7053-908f-46d0-8b5e-f3fdb57d433e","Type":"ContainerStarted","Data":"52d8cc0e7ab436824c54a220394ae05bc23c9b9a1945d7923f063bdae30c57a1"}
Nov 03 09:15:02.216379 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:02.216358    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894955-7v5xg" event={"ID":"750847c9-fd9f-4d39-8705-ccea1fd3fc17","Type":"ContainerStarted","Data":"963726784f0917ed6639921a4e9db3d6ba213b7d8755baf92457dde6f198ae5b"}
Nov 03 09:15:02.216561 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:02.216542    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894955-7v5xg" event={"ID":"750847c9-fd9f-4d39-8705-ccea1fd3fc17","Type":"ContainerStarted","Data":"8ab98ce00b9a9b22f271e850dee6ec5222c515c1ce414bb85ffb93b3acef8af8"}
Nov 03 09:15:02.222111 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:02.222088    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894955-z25mc" event={"ID":"4cca539f-d6ea-4402-ac67-28886b17c397","Type":"ContainerStarted","Data":"d147f361048a4422c30e26bfd2e71bd6250734c66a1623be29a47ff6159450ad"}
Nov 03 09:15:02.222263 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:02.222239    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894955-z25mc" event={"ID":"4cca539f-d6ea-4402-ac67-28886b17c397","Type":"ContainerStarted","Data":"d7a68b843cb633d5125e791691df66666b8d2d9c205d67288e682275643817b6"}
Nov 03 09:15:02.225200 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:02.225173    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-121ee6-29894955-6nz68" event={"ID":"00db03c4-9076-4e59-a799-a260f34a1e7d","Type":"ContainerStarted","Data":"08553f3d7ee3800c75cff4e72bd378d02b2d880644b8aeb794ba34f2140cae5e"}
Nov 03 09:15:02.225425 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:02.225220    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-121ee6-29894955-6nz68" event={"ID":"00db03c4-9076-4e59-a799-a260f34a1e7d","Type":"ContainerStarted","Data":"8ca8be60c5639ee27c7eeeb98ac760683b50a94319de78e73c422094c58ecd04"}
Nov 03 09:15:02.328733 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:02.328658    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894955-58wlg" podStartSLOduration=2.328638808 podStartE2EDuration="2.328638808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:15:02.326249698 +0000 UTC m=+2126.595313166" watchObservedRunningTime="2026-11-03 09:15:02.328638808 +0000 UTC m=+2126.597702243"
Nov 03 09:15:02.443756 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:02.443713    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-121ee6-29894955-6nz68" podStartSLOduration=2.443699016 podStartE2EDuration="2.443699016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:15:02.441791628 +0000 UTC m=+2126.710855073" watchObservedRunningTime="2026-11-03 09:15:02.443699016 +0000 UTC m=+2126.712762449"
Nov 03 09:15:02.558396 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:02.558344    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894955-z25mc" podStartSLOduration=2.558328842 podStartE2EDuration="2.558328842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:15:02.556050593 +0000 UTC m=+2126.825114053" watchObservedRunningTime="2026-11-03 09:15:02.558328842 +0000 UTC m=+2126.827392325"
Nov 03 09:15:02.673254 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:02.673198    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-133b31-29894955-wtst5" podStartSLOduration=2.673182907 podStartE2EDuration="2.673182907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:15:02.671116643 +0000 UTC m=+2126.940180110" watchObservedRunningTime="2026-11-03 09:15:02.673182907 +0000 UTC m=+2126.942246340"
Nov 03 09:15:02.787693 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:02.787593    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0eecfd-76cbfb4455-4ztkx" podStartSLOduration=2.787578505 podStartE2EDuration="2.787578505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:15:02.786626779 +0000 UTC m=+2127.055690229" watchObservedRunningTime="2026-11-03 09:15:02.787578505 +0000 UTC m=+2127.056641940"
Nov 03 09:15:03.047773 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:03.047446    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894955-7v5xg" podStartSLOduration=3.047423522 podStartE2EDuration="3.047423522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:15:02.930718008 +0000 UTC m=+2127.199781444" watchObservedRunningTime="2026-11-03 09:15:03.047423522 +0000 UTC m=+2127.316486957"
Nov 03 09:15:03.048502 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:03.048466    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894955-777bf" podStartSLOduration=3.048453119 podStartE2EDuration="3.048453119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:15:03.047517587 +0000 UTC m=+2127.316581040" watchObservedRunningTime="2026-11-03 09:15:03.048453119 +0000 UTC m=+2127.317516552"
Nov 03 09:15:03.165856 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:03.165805    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04934e-29894955-2z4nq" podStartSLOduration=3.165789796 podStartE2EDuration="3.165789796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:15:03.162897922 +0000 UTC m=+2127.431961357" watchObservedRunningTime="2026-11-03 09:15:03.165789796 +0000 UTC m=+2127.434853274"
Nov 03 09:15:03.232573 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:03.232543    4473 generic.go:356] "Generic (PLEG): container finished" podID="750847c9-fd9f-4d39-8705-ccea1fd3fc17" containerID="963726784f0917ed6639921a4e9db3d6ba213b7d8755baf92457dde6f198ae5b" exitCode=60
Nov 03 09:15:03.232759 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:03.232629    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894955-7v5xg" event={"ID":"750847c9-fd9f-4d39-8705-ccea1fd3fc17","Type":"ContainerDied","Data":"963726784f0917ed6639921a4e9db3d6ba213b7d8755baf92457dde6f198ae5b"}
Nov 03 09:15:03.232925 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:03.232908    4473 scope.go:122] "RemoveContainer" containerID="963726784f0917ed6639921a4e9db3d6ba213b7d8755baf92457dde6f198ae5b"
Nov 03 09:15:03.234381 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:03.234348    4473 generic.go:356] "Generic (PLEG): container finished" podID="4cca539f-d6ea-4402-ac67-28886b17c397" containerID="d147f361048a4422c30e26bfd2e71bd6250734c66a1623be29a47ff6159450ad" exitCode=60
Nov 03 09:15:03.234489 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:03.234377    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894955-z25mc" event={"ID":"4cca539f-d6ea-4402-ac67-28886b17c397","Type":"ContainerDied","Data":"d147f361048a4422c30e26bfd2e71bd6250734c66a1623be29a47ff6159450ad"}
Nov 03 09:15:03.234657 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:03.234640    4473 scope.go:122] "RemoveContainer" containerID="d147f361048a4422c30e26bfd2e71bd6250734c66a1623be29a47ff6159450ad"
Nov 03 09:15:03.239441 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:03.236338    4473 generic.go:356] "Generic (PLEG): container finished" podID="00db03c4-9076-4e59-a799-a260f34a1e7d" containerID="08553f3d7ee3800c75cff4e72bd378d02b2d880644b8aeb794ba34f2140cae5e" exitCode=60
Nov 03 09:15:03.239441 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:03.236405    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-121ee6-29894955-6nz68" event={"ID":"00db03c4-9076-4e59-a799-a260f34a1e7d","Type":"ContainerDied","Data":"08553f3d7ee3800c75cff4e72bd378d02b2d880644b8aeb794ba34f2140cae5e"}
Nov 03 09:15:03.239441 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:03.236580    4473 scope.go:122] "RemoveContainer" containerID="08553f3d7ee3800c75cff4e72bd378d02b2d880644b8aeb794ba34f2140cae5e"
Nov 03 09:15:03.239441 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:03.238125    4473 generic.go:356] "Generic (PLEG): container finished" podID="61f3f0ff-1cbf-4e08-8d44-f44952e3b01c" containerID="13b30ed6d5e49d84baeb66a44985c26256e2cff7e495fe4e62f20f3977d2416d" exitCode=60
Nov 03 09:15:03.239441 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:03.238196    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894955-58wlg" event={"ID":"61f3f0ff-1cbf-4e08-8d44-f44952e3b01c","Type":"ContainerDied","Data":"13b30ed6d5e49d84baeb66a44985c26256e2cff7e495fe4e62f20f3977d2416d"}
Nov 03 09:15:03.239441 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:03.238353    4473 scope.go:122] "RemoveContainer" containerID="13b30ed6d5e49d84baeb66a44985c26256e2cff7e495fe4e62f20f3977d2416d"
Nov 03 09:15:03.239849 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:03.239830    4473 generic.go:356] "Generic (PLEG): container finished" podID="bf19596b-0d54-4f5f-a6d9-433b4c130cbc" containerID="b98125884ecb604c97dac1319f68641cdb1303cb15640f96096ec9e54a1e2d42" exitCode=60
Nov 03 09:15:03.239926 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:03.239889    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-133b31-29894955-wtst5" event={"ID":"bf19596b-0d54-4f5f-a6d9-433b4c130cbc","Type":"ContainerDied","Data":"b98125884ecb604c97dac1319f68641cdb1303cb15640f96096ec9e54a1e2d42"}
Nov 03 09:15:03.240107 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:03.240088    4473 scope.go:122] "RemoveContainer" containerID="b98125884ecb604c97dac1319f68641cdb1303cb15640f96096ec9e54a1e2d42"
Nov 03 09:15:03.241512 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:03.241493    4473 generic.go:356] "Generic (PLEG): container finished" podID="cb6573f4-bd1c-4383-bb08-7fdae8f679d2" containerID="fd3d356cac2a425d55f89fa691bef8f592511cfcc5fca1d708f3e76476482680" exitCode=60
Nov 03 09:15:03.241606 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:03.241559    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04934e-29894955-2z4nq" event={"ID":"cb6573f4-bd1c-4383-bb08-7fdae8f679d2","Type":"ContainerDied","Data":"fd3d356cac2a425d55f89fa691bef8f592511cfcc5fca1d708f3e76476482680"}
Nov 03 09:15:03.241766 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:03.241747    4473 scope.go:122] "RemoveContainer" containerID="fd3d356cac2a425d55f89fa691bef8f592511cfcc5fca1d708f3e76476482680"
Nov 03 09:15:03.248573 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:03.248552    4473 generic.go:356] "Generic (PLEG): container finished" podID="20ec7053-908f-46d0-8b5e-f3fdb57d433e" containerID="223b897638e0503d3beabfaa14eca6c0861a88491b39fee5b10cbdf00cf8b88d" exitCode=60
Nov 03 09:15:03.249365 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:03.249344    4473 scope.go:122] "RemoveContainer" containerID="223b897638e0503d3beabfaa14eca6c0861a88491b39fee5b10cbdf00cf8b88d"
Nov 03 09:15:03.249474 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:03.249378    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894955-777bf" event={"ID":"20ec7053-908f-46d0-8b5e-f3fdb57d433e","Type":"ContainerDied","Data":"223b897638e0503d3beabfaa14eca6c0861a88491b39fee5b10cbdf00cf8b88d"}
Nov 03 09:15:04.267857 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:04.267764    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894955-z25mc" event={"ID":"4cca539f-d6ea-4402-ac67-28886b17c397","Type":"ContainerStarted","Data":"7ab668affeebf7524d6a00f28621e135f613fc908f8e64f67b2ce46411d3b35b"}
Nov 03 09:15:04.269782 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:04.269749    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-121ee6-29894955-6nz68" event={"ID":"00db03c4-9076-4e59-a799-a260f34a1e7d","Type":"ContainerStarted","Data":"f547a2110044b1bc62c502c29c594382c2122e6d86ed26e23ee3cc63dc742edb"}
Nov 03 09:15:04.271774 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:04.271752    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894955-58wlg" event={"ID":"61f3f0ff-1cbf-4e08-8d44-f44952e3b01c","Type":"ContainerStarted","Data":"0f8d8cac4f1a99ee25b0aeac36db83fed94ef94302ae6a44180a55dfcfb7724f"}
Nov 03 09:15:04.274098 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:04.274070    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-133b31-29894955-wtst5" event={"ID":"bf19596b-0d54-4f5f-a6d9-433b4c130cbc","Type":"ContainerStarted","Data":"c33b7894dcb2e44341fc6a1175d8bfb5cf5e1d95395d4036293d597865e24f7c"}
Nov 03 09:15:04.276191 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:04.276165    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04934e-29894955-2z4nq" event={"ID":"cb6573f4-bd1c-4383-bb08-7fdae8f679d2","Type":"ContainerStarted","Data":"ce2b008b04743b6324aabdd4611465d35e20131bd43d98f19c29b87786fb3e34"}
Nov 03 09:15:04.278117 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:04.278093    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894955-777bf" event={"ID":"20ec7053-908f-46d0-8b5e-f3fdb57d433e","Type":"ContainerStarted","Data":"3a89dec0928f515a50aa52518c6cfba66dab3274427d9c6f9026cb31fc1e34f6"}
Nov 03 09:15:04.280000 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:04.279978    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894955-7v5xg" event={"ID":"750847c9-fd9f-4d39-8705-ccea1fd3fc17","Type":"ContainerStarted","Data":"7bdb2ee344359679c0d97b2a1cd6f52314c83a28252f54658b12879f412161b2"}
Nov 03 09:15:05.287876 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:05.287841    4473 generic.go:356] "Generic (PLEG): container finished" podID="bf19596b-0d54-4f5f-a6d9-433b4c130cbc" containerID="c33b7894dcb2e44341fc6a1175d8bfb5cf5e1d95395d4036293d597865e24f7c" exitCode=60
Nov 03 09:15:05.288360 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:05.287881    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-133b31-29894955-wtst5" event={"ID":"bf19596b-0d54-4f5f-a6d9-433b4c130cbc","Type":"ContainerDied","Data":"c33b7894dcb2e44341fc6a1175d8bfb5cf5e1d95395d4036293d597865e24f7c"}
Nov 03 09:15:05.288360 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:05.287938    4473 scope.go:122] "RemoveContainer" containerID="b98125884ecb604c97dac1319f68641cdb1303cb15640f96096ec9e54a1e2d42"
Nov 03 09:15:05.288360 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:05.288243    4473 scope.go:122] "RemoveContainer" containerID="c33b7894dcb2e44341fc6a1175d8bfb5cf5e1d95395d4036293d597865e24f7c"
Nov 03 09:15:05.288581 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:15:05.288556    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-133b31-29894955-wtst5_odh-ai-gateway-infra(bf19596b-0d54-4f5f-a6d9-433b4c130cbc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-133b31-29894955-wtst5" podUID="bf19596b-0d54-4f5f-a6d9-433b4c130cbc"
Nov 03 09:15:05.289757 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:05.289740    4473 generic.go:356] "Generic (PLEG): container finished" podID="cb6573f4-bd1c-4383-bb08-7fdae8f679d2" containerID="ce2b008b04743b6324aabdd4611465d35e20131bd43d98f19c29b87786fb3e34" exitCode=60
Nov 03 09:15:05.289850 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:05.289802    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04934e-29894955-2z4nq" event={"ID":"cb6573f4-bd1c-4383-bb08-7fdae8f679d2","Type":"ContainerDied","Data":"ce2b008b04743b6324aabdd4611465d35e20131bd43d98f19c29b87786fb3e34"}
Nov 03 09:15:05.290144 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:05.290126    4473 scope.go:122] "RemoveContainer" containerID="ce2b008b04743b6324aabdd4611465d35e20131bd43d98f19c29b87786fb3e34"
Nov 03 09:15:05.290394 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:15:05.290375    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-04934e-29894955-2z4nq_odh-ai-gateway-infra(cb6573f4-bd1c-4383-bb08-7fdae8f679d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04934e-29894955-2z4nq" podUID="cb6573f4-bd1c-4383-bb08-7fdae8f679d2"
Nov 03 09:15:05.291687 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:05.291668    4473 generic.go:356] "Generic (PLEG): container finished" podID="20ec7053-908f-46d0-8b5e-f3fdb57d433e" containerID="3a89dec0928f515a50aa52518c6cfba66dab3274427d9c6f9026cb31fc1e34f6" exitCode=60
Nov 03 09:15:05.291789 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:05.291753    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894955-777bf" event={"ID":"20ec7053-908f-46d0-8b5e-f3fdb57d433e","Type":"ContainerDied","Data":"3a89dec0928f515a50aa52518c6cfba66dab3274427d9c6f9026cb31fc1e34f6"}
Nov 03 09:15:05.291976 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:05.291960    4473 scope.go:122] "RemoveContainer" containerID="3a89dec0928f515a50aa52518c6cfba66dab3274427d9c6f9026cb31fc1e34f6"
Nov 03 09:15:05.292241 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:15:05.292225    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-26e861-29894955-777bf_odh-ai-gateway-infra(20ec7053-908f-46d0-8b5e-f3fdb57d433e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894955-777bf" podUID="20ec7053-908f-46d0-8b5e-f3fdb57d433e"
Nov 03 09:15:05.293784 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:05.293766    4473 generic.go:356] "Generic (PLEG): container finished" podID="750847c9-fd9f-4d39-8705-ccea1fd3fc17" containerID="7bdb2ee344359679c0d97b2a1cd6f52314c83a28252f54658b12879f412161b2" exitCode=60
Nov 03 09:15:05.293879 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:05.293826    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894955-7v5xg" event={"ID":"750847c9-fd9f-4d39-8705-ccea1fd3fc17","Type":"ContainerDied","Data":"7bdb2ee344359679c0d97b2a1cd6f52314c83a28252f54658b12879f412161b2"}
Nov 03 09:15:05.294093 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:05.294073    4473 scope.go:122] "RemoveContainer" containerID="7bdb2ee344359679c0d97b2a1cd6f52314c83a28252f54658b12879f412161b2"
Nov 03 09:15:05.294317 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:15:05.294296    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-77207e-29894955-7v5xg_odh-ai-gateway-infra(750847c9-fd9f-4d39-8705-ccea1fd3fc17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894955-7v5xg" podUID="750847c9-fd9f-4d39-8705-ccea1fd3fc17"
Nov 03 09:15:05.295843 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:05.295812    4473 generic.go:356] "Generic (PLEG): container finished" podID="4cca539f-d6ea-4402-ac67-28886b17c397" containerID="7ab668affeebf7524d6a00f28621e135f613fc908f8e64f67b2ce46411d3b35b" exitCode=60
Nov 03 09:15:05.295978 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:05.295875    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894955-z25mc" event={"ID":"4cca539f-d6ea-4402-ac67-28886b17c397","Type":"ContainerDied","Data":"7ab668affeebf7524d6a00f28621e135f613fc908f8e64f67b2ce46411d3b35b"}
Nov 03 09:15:05.296170 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:05.296155    4473 scope.go:122] "RemoveContainer" containerID="7ab668affeebf7524d6a00f28621e135f613fc908f8e64f67b2ce46411d3b35b"
Nov 03 09:15:05.296377 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:15:05.296361    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-910d50-29894955-z25mc_odh-ai-gateway-infra(4cca539f-d6ea-4402-ac67-28886b17c397)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894955-z25mc" podUID="4cca539f-d6ea-4402-ac67-28886b17c397"
Nov 03 09:15:05.298103 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:05.298082    4473 generic.go:356] "Generic (PLEG): container finished" podID="00db03c4-9076-4e59-a799-a260f34a1e7d" containerID="f547a2110044b1bc62c502c29c594382c2122e6d86ed26e23ee3cc63dc742edb" exitCode=60
Nov 03 09:15:05.298195 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:05.298113    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-121ee6-29894955-6nz68" event={"ID":"00db03c4-9076-4e59-a799-a260f34a1e7d","Type":"ContainerDied","Data":"f547a2110044b1bc62c502c29c594382c2122e6d86ed26e23ee3cc63dc742edb"}
Nov 03 09:15:05.298372 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:05.298354    4473 scope.go:122] "RemoveContainer" containerID="f547a2110044b1bc62c502c29c594382c2122e6d86ed26e23ee3cc63dc742edb"
Nov 03 09:15:05.298610 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:15:05.298588    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-121ee6-29894955-6nz68_odh-ai-gateway-infra(00db03c4-9076-4e59-a799-a260f34a1e7d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-121ee6-29894955-6nz68" podUID="00db03c4-9076-4e59-a799-a260f34a1e7d"
Nov 03 09:15:05.300227 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:05.300205    4473 generic.go:356] "Generic (PLEG): container finished" podID="61f3f0ff-1cbf-4e08-8d44-f44952e3b01c" containerID="0f8d8cac4f1a99ee25b0aeac36db83fed94ef94302ae6a44180a55dfcfb7724f" exitCode=60
Nov 03 09:15:05.300300 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:05.300266    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894955-58wlg" event={"ID":"61f3f0ff-1cbf-4e08-8d44-f44952e3b01c","Type":"ContainerDied","Data":"0f8d8cac4f1a99ee25b0aeac36db83fed94ef94302ae6a44180a55dfcfb7724f"}
Nov 03 09:15:05.300419 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:05.300406    4473 scope.go:122] "RemoveContainer" containerID="0f8d8cac4f1a99ee25b0aeac36db83fed94ef94302ae6a44180a55dfcfb7724f"
Nov 03 09:15:05.300600 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:15:05.300583    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-18f9da-29894955-58wlg_odh-ai-gateway-infra(61f3f0ff-1cbf-4e08-8d44-f44952e3b01c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894955-58wlg" podUID="61f3f0ff-1cbf-4e08-8d44-f44952e3b01c"
Nov 03 09:15:05.301584 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:05.301565    4473 scope.go:122] "RemoveContainer" containerID="fd3d356cac2a425d55f89fa691bef8f592511cfcc5fca1d708f3e76476482680"
Nov 03 09:15:05.314663 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:05.314644    4473 scope.go:122] "RemoveContainer" containerID="223b897638e0503d3beabfaa14eca6c0861a88491b39fee5b10cbdf00cf8b88d"
Nov 03 09:15:05.327743 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:05.327718    4473 scope.go:122] "RemoveContainer" containerID="963726784f0917ed6639921a4e9db3d6ba213b7d8755baf92457dde6f198ae5b"
Nov 03 09:15:05.341460 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:05.341433    4473 scope.go:122] "RemoveContainer" containerID="d147f361048a4422c30e26bfd2e71bd6250734c66a1623be29a47ff6159450ad"
Nov 03 09:15:05.354532 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:05.354513    4473 scope.go:122] "RemoveContainer" containerID="08553f3d7ee3800c75cff4e72bd378d02b2d880644b8aeb794ba34f2140cae5e"
Nov 03 09:15:05.367797 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:05.367778    4473 scope.go:122] "RemoveContainer" containerID="13b30ed6d5e49d84baeb66a44985c26256e2cff7e495fe4e62f20f3977d2416d"
Nov 03 09:15:06.308363 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:06.308327    4473 scope.go:122] "RemoveContainer" containerID="c33b7894dcb2e44341fc6a1175d8bfb5cf5e1d95395d4036293d597865e24f7c"
Nov 03 09:15:06.308848 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:15:06.308684    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-133b31-29894955-wtst5_odh-ai-gateway-infra(bf19596b-0d54-4f5f-a6d9-433b4c130cbc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-133b31-29894955-wtst5" podUID="bf19596b-0d54-4f5f-a6d9-433b4c130cbc"
Nov 03 09:15:06.309983 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:06.309960    4473 scope.go:122] "RemoveContainer" containerID="ce2b008b04743b6324aabdd4611465d35e20131bd43d98f19c29b87786fb3e34"
Nov 03 09:15:06.310244 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:15:06.310219    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-04934e-29894955-2z4nq_odh-ai-gateway-infra(cb6573f4-bd1c-4383-bb08-7fdae8f679d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04934e-29894955-2z4nq" podUID="cb6573f4-bd1c-4383-bb08-7fdae8f679d2"
Nov 03 09:15:06.311422 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:06.311403    4473 scope.go:122] "RemoveContainer" containerID="3a89dec0928f515a50aa52518c6cfba66dab3274427d9c6f9026cb31fc1e34f6"
Nov 03 09:15:06.311673 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:15:06.311650    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-26e861-29894955-777bf_odh-ai-gateway-infra(20ec7053-908f-46d0-8b5e-f3fdb57d433e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894955-777bf" podUID="20ec7053-908f-46d0-8b5e-f3fdb57d433e"
Nov 03 09:15:06.312920 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:06.312900    4473 scope.go:122] "RemoveContainer" containerID="7bdb2ee344359679c0d97b2a1cd6f52314c83a28252f54658b12879f412161b2"
Nov 03 09:15:06.313175 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:15:06.313156    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-77207e-29894955-7v5xg_odh-ai-gateway-infra(750847c9-fd9f-4d39-8705-ccea1fd3fc17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894955-7v5xg" podUID="750847c9-fd9f-4d39-8705-ccea1fd3fc17"
Nov 03 09:15:06.314343 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:06.314324    4473 scope.go:122] "RemoveContainer" containerID="7ab668affeebf7524d6a00f28621e135f613fc908f8e64f67b2ce46411d3b35b"
Nov 03 09:15:06.314531 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:15:06.314514    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-910d50-29894955-z25mc_odh-ai-gateway-infra(4cca539f-d6ea-4402-ac67-28886b17c397)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894955-z25mc" podUID="4cca539f-d6ea-4402-ac67-28886b17c397"
Nov 03 09:15:06.315958 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:06.315925    4473 scope.go:122] "RemoveContainer" containerID="f547a2110044b1bc62c502c29c594382c2122e6d86ed26e23ee3cc63dc742edb"
Nov 03 09:15:06.316157 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:15:06.316139    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-121ee6-29894955-6nz68_odh-ai-gateway-infra(00db03c4-9076-4e59-a799-a260f34a1e7d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-121ee6-29894955-6nz68" podUID="00db03c4-9076-4e59-a799-a260f34a1e7d"
Nov 03 09:15:06.317660 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:06.317644    4473 scope.go:122] "RemoveContainer" containerID="0f8d8cac4f1a99ee25b0aeac36db83fed94ef94302ae6a44180a55dfcfb7724f"
Nov 03 09:15:06.317817 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:15:06.317801    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-18f9da-29894955-58wlg_odh-ai-gateway-infra(61f3f0ff-1cbf-4e08-8d44-f44952e3b01c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894955-58wlg" podUID="61f3f0ff-1cbf-4e08-8d44-f44952e3b01c"
Nov 03 09:15:08.937558 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:08.937523    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04934e-29894955-2z4nq"]
Nov 03 09:15:09.198467 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:09.198405    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04934e-29894955-2z4nq"
Nov 03 09:15:09.215641 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:09.215615    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-04934e-548676699b-sctv8"]
Nov 03 09:15:09.215791 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:09.215773    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-04934e-548676699b-sctv8" podUID="d59877f4-7a69-496e-b04b-7c3df4c9a2c4" containerName="payload-processing" containerID="cri-o://6a0e54fe21e8e0d65c0b955489b981812406788c38cc82877067734616e76075" gracePeriod=30
Nov 03 09:15:09.324382 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:09.324345    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb6573f4-bd1c-4383-bb08-7fdae8f679d2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb6573f4-bd1c-4383-bb08-7fdae8f679d2-maas-api-service-ca\") pod \"cb6573f4-bd1c-4383-bb08-7fdae8f679d2\" (UID: \"cb6573f4-bd1c-4383-bb08-7fdae8f679d2\") "
Nov 03 09:15:09.324653 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:09.324632    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb6573f4-bd1c-4383-bb08-7fdae8f679d2-kube-api-access-p2rc2\" (UniqueName: \"kubernetes.io/projected/cb6573f4-bd1c-4383-bb08-7fdae8f679d2-kube-api-access-p2rc2\") pod \"cb6573f4-bd1c-4383-bb08-7fdae8f679d2\" (UID: \"cb6573f4-bd1c-4383-bb08-7fdae8f679d2\") "
Nov 03 09:15:09.329865 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:09.329643    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb6573f4-bd1c-4383-bb08-7fdae8f679d2-kube-api-access-p2rc2" pod "cb6573f4-bd1c-4383-bb08-7fdae8f679d2" (UID: "cb6573f4-bd1c-4383-bb08-7fdae8f679d2"). InnerVolumeSpecName "kube-api-access-p2rc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:15:09.329865 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:09.329733    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb6573f4-bd1c-4383-bb08-7fdae8f679d2-maas-api-service-ca" pod "cb6573f4-bd1c-4383-bb08-7fdae8f679d2" (UID: "cb6573f4-bd1c-4383-bb08-7fdae8f679d2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:15:09.356504 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:09.356466    4473 generic.go:356] "Generic (PLEG): container finished" podID="d59877f4-7a69-496e-b04b-7c3df4c9a2c4" containerID="6a0e54fe21e8e0d65c0b955489b981812406788c38cc82877067734616e76075" exitCode=0
Nov 03 09:15:09.356603 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:09.356536    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-04934e-548676699b-sctv8" event={"ID":"d59877f4-7a69-496e-b04b-7c3df4c9a2c4","Type":"ContainerDied","Data":"6a0e54fe21e8e0d65c0b955489b981812406788c38cc82877067734616e76075"}
Nov 03 09:15:09.358488 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:09.358462    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04934e-29894955-2z4nq" event={"ID":"cb6573f4-bd1c-4383-bb08-7fdae8f679d2","Type":"ContainerDied","Data":"6fa99d368ee46c3a362849f05a120d2c17ae6939abfc80695dc3bfce94f0ddaa"}
Nov 03 09:15:09.358592 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:09.358523    4473 scope.go:122] "RemoveContainer" containerID="ce2b008b04743b6324aabdd4611465d35e20131bd43d98f19c29b87786fb3e34"
Nov 03 09:15:09.358592 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:09.358534    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04934e-29894955-2z4nq"
Nov 03 09:15:09.427443 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:09.427415    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb6573f4-bd1c-4383-bb08-7fdae8f679d2-maas-api-service-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:09.427443 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:09.427443    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2rc2\" (UniqueName: \"kubernetes.io/projected/cb6573f4-bd1c-4383-bb08-7fdae8f679d2-kube-api-access-p2rc2\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:09.503331 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:09.503309    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-04934e-548676699b-sctv8"
Nov 03 09:15:09.546911 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:09.546883    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04934e-29894955-2z4nq"]
Nov 03 09:15:09.569971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:09.554460    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-04934e-29894955-2z4nq"]
Nov 03 09:15:09.629333 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:09.629294    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d59877f4-7a69-496e-b04b-7c3df4c9a2c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d59877f4-7a69-496e-b04b-7c3df4c9a2c4-plugins-config-volume\") pod \"d59877f4-7a69-496e-b04b-7c3df4c9a2c4\" (UID: \"d59877f4-7a69-496e-b04b-7c3df4c9a2c4\") "
Nov 03 09:15:09.629524 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:09.629379    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d59877f4-7a69-496e-b04b-7c3df4c9a2c4-kube-api-access-pnd77\" (UniqueName: \"kubernetes.io/projected/d59877f4-7a69-496e-b04b-7c3df4c9a2c4-kube-api-access-pnd77\") pod \"d59877f4-7a69-496e-b04b-7c3df4c9a2c4\" (UID: \"d59877f4-7a69-496e-b04b-7c3df4c9a2c4\") "
Nov 03 09:15:09.629712 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:09.629688    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d59877f4-7a69-496e-b04b-7c3df4c9a2c4-plugins-config-volume" pod "d59877f4-7a69-496e-b04b-7c3df4c9a2c4" (UID: "d59877f4-7a69-496e-b04b-7c3df4c9a2c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:15:09.629979 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:09.629936    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d59877f4-7a69-496e-b04b-7c3df4c9a2c4-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:09.631858 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:09.631833    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d59877f4-7a69-496e-b04b-7c3df4c9a2c4-kube-api-access-pnd77" pod "d59877f4-7a69-496e-b04b-7c3df4c9a2c4" (UID: "d59877f4-7a69-496e-b04b-7c3df4c9a2c4"). InnerVolumeSpecName "kube-api-access-pnd77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:15:09.731217 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:09.731144    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnd77\" (UniqueName: \"kubernetes.io/projected/d59877f4-7a69-496e-b04b-7c3df4c9a2c4-kube-api-access-pnd77\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:09.886005 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:09.885971    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb6573f4-bd1c-4383-bb08-7fdae8f679d2" path="/var/lib/kubelet/pods/cb6573f4-bd1c-4383-bb08-7fdae8f679d2/volumes"
Nov 03 09:15:10.376110 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:10.374868    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-04934e-548676699b-sctv8" event={"ID":"d59877f4-7a69-496e-b04b-7c3df4c9a2c4","Type":"ContainerDied","Data":"859a13a17e1b14cc5a044de71a84f5299096273536e62d902f219e55b18f65e1"}
Nov 03 09:15:10.376110 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:10.374940    4473 scope.go:122] "RemoveContainer" containerID="6a0e54fe21e8e0d65c0b955489b981812406788c38cc82877067734616e76075"
Nov 03 09:15:10.376110 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:10.375074    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-04934e-548676699b-sctv8"
Nov 03 09:15:10.589311 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:10.587574    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-04934e-548676699b-sctv8"]
Nov 03 09:15:10.611986 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:10.608484    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-04934e-548676699b-sctv8"]
Nov 03 09:15:11.870243 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:11.870201    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d59877f4-7a69-496e-b04b-7c3df4c9a2c4" path="/var/lib/kubelet/pods/d59877f4-7a69-496e-b04b-7c3df4c9a2c4/volumes"
Nov 03 09:15:13.252645 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:13.252620    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0eecfd-76cbfb4455-4ztkx"
Nov 03 09:15:13.995528 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:13.995487    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c826a7d-47qmk"]
Nov 03 09:15:13.996139 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:13.996119    4473 cpu_manager.go:414] "containerMap: removing container" podUID="d59877f4-7a69-496e-b04b-7c3df4c9a2c4" containerName="payload-processing"
Nov 03 09:15:13.996265 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:13.996140    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d59877f4-7a69-496e-b04b-7c3df4c9a2c4" containerName="payload-processing"
Nov 03 09:15:13.996265 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:13.996157    4473 cpu_manager.go:414] "containerMap: removing container" podUID="cb6573f4-bd1c-4383-bb08-7fdae8f679d2" containerName="cleanup"
Nov 03 09:15:13.996265 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:13.996165    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb6573f4-bd1c-4383-bb08-7fdae8f679d2" containerName="cleanup"
Nov 03 09:15:13.996265 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:13.996174    4473 cpu_manager.go:414] "containerMap: removing container" podUID="cb6573f4-bd1c-4383-bb08-7fdae8f679d2" containerName="cleanup"
Nov 03 09:15:13.996265 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:13.996181    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb6573f4-bd1c-4383-bb08-7fdae8f679d2" containerName="cleanup"
Nov 03 09:15:13.996521 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:13.996393    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d59877f4-7a69-496e-b04b-7c3df4c9a2c4" containerName="payload-processing"
Nov 03 09:15:13.996521 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:13.996408    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb6573f4-bd1c-4383-bb08-7fdae8f679d2" containerName="cleanup"
Nov 03 09:15:13.996521 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:13.996418    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb6573f4-bd1c-4383-bb08-7fdae8f679d2" containerName="cleanup"
Nov 03 09:15:14.001079 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:14.001059    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c826a7d-47qmk"
Nov 03 09:15:14.032657 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:14.032631    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17acd40d-9a11-4a0e-9a09-88f9756e2402-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c826a7d-47qmk\" (UID: \"17acd40d-9a11-4a0e-9a09-88f9756e2402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c826a7d-47qmk"
Nov 03 09:15:14.032797 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:14.032760    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhp27\" (UniqueName: \"kubernetes.io/projected/17acd40d-9a11-4a0e-9a09-88f9756e2402-kube-api-access-vhp27\") pod \"maas-api-revoke-sub-7c826a7d-47qmk\" (UID: \"17acd40d-9a11-4a0e-9a09-88f9756e2402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c826a7d-47qmk"
Nov 03 09:15:14.032797 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:14.032787    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17acd40d-9a11-4a0e-9a09-88f9756e2402-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c826a7d-47qmk\" (UID: \"17acd40d-9a11-4a0e-9a09-88f9756e2402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c826a7d-47qmk"
Nov 03 09:15:14.055821 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:14.055799    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:15:14.110190 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:14.110160    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c826a7d-47qmk"]
Nov 03 09:15:14.134597 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:14.134568    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17acd40d-9a11-4a0e-9a09-88f9756e2402-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c826a7d-47qmk\" (UID: \"17acd40d-9a11-4a0e-9a09-88f9756e2402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c826a7d-47qmk"
Nov 03 09:15:14.134739 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:14.134680    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhp27\" (UniqueName: \"kubernetes.io/projected/17acd40d-9a11-4a0e-9a09-88f9756e2402-kube-api-access-vhp27\") pod \"maas-api-revoke-sub-7c826a7d-47qmk\" (UID: \"17acd40d-9a11-4a0e-9a09-88f9756e2402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c826a7d-47qmk"
Nov 03 09:15:14.134739 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:14.134718    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17acd40d-9a11-4a0e-9a09-88f9756e2402-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c826a7d-47qmk\" (UID: \"17acd40d-9a11-4a0e-9a09-88f9756e2402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c826a7d-47qmk"
Nov 03 09:15:14.135424 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:14.135398    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17acd40d-9a11-4a0e-9a09-88f9756e2402-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c826a7d-47qmk\" (UID: \"17acd40d-9a11-4a0e-9a09-88f9756e2402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c826a7d-47qmk"
Nov 03 09:15:14.137254 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:14.137235    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17acd40d-9a11-4a0e-9a09-88f9756e2402-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c826a7d-47qmk\" (UID: \"17acd40d-9a11-4a0e-9a09-88f9756e2402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c826a7d-47qmk"
Nov 03 09:15:14.193736 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:14.193711    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhp27\" (UniqueName: \"kubernetes.io/projected/17acd40d-9a11-4a0e-9a09-88f9756e2402-kube-api-access-vhp27\") pod \"maas-api-revoke-sub-7c826a7d-47qmk\" (UID: \"17acd40d-9a11-4a0e-9a09-88f9756e2402\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c826a7d-47qmk"
Nov 03 09:15:14.314834 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:14.314749    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c826a7d-47qmk"
Nov 03 09:15:14.691606 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:14.691583    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c826a7d-47qmk"]
Nov 03 09:15:14.692764 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:15:14.692740    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17acd40d_9a11_4a0e_9a09_88f9756e2402.slice/crio-bc7bec9bda92132dd7b157202c929fd39c388f0c008e3b3c4cf35c9d7622d0d4 WatchSource:0}: Error finding container bc7bec9bda92132dd7b157202c929fd39c388f0c008e3b3c4cf35c9d7622d0d4: Status 404 returned error can't find the container with id bc7bec9bda92132dd7b157202c929fd39c388f0c008e3b3c4cf35c9d7622d0d4
Nov 03 09:15:14.863387 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:14.863356    4473 scope.go:122] "RemoveContainer" containerID="f547a2110044b1bc62c502c29c594382c2122e6d86ed26e23ee3cc63dc742edb"
Nov 03 09:15:14.863545 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:14.863451    4473 scope.go:122] "RemoveContainer" containerID="7bdb2ee344359679c0d97b2a1cd6f52314c83a28252f54658b12879f412161b2"
Nov 03 09:15:15.411004 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:15.410964    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c826a7d-47qmk" event={"ID":"17acd40d-9a11-4a0e-9a09-88f9756e2402","Type":"ContainerStarted","Data":"bb84b2efe161ca4b53b561e03a8d54f19042871f0233bbf499a0a692dc863043"}
Nov 03 09:15:15.411456 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:15.411022    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c826a7d-47qmk" event={"ID":"17acd40d-9a11-4a0e-9a09-88f9756e2402","Type":"ContainerStarted","Data":"bc7bec9bda92132dd7b157202c929fd39c388f0c008e3b3c4cf35c9d7622d0d4"}
Nov 03 09:15:15.413139 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:15.413107    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894955-7v5xg" event={"ID":"750847c9-fd9f-4d39-8705-ccea1fd3fc17","Type":"ContainerStarted","Data":"d792438f2c04965b9ad4ffcbc69ac45726e4ed5cc8e95e7bd8b5e81dce1d44d8"}
Nov 03 09:15:15.415140 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:15.415119    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-121ee6-29894955-6nz68" event={"ID":"00db03c4-9076-4e59-a799-a260f34a1e7d","Type":"ContainerStarted","Data":"189d5f1a2b5c31b54cf32b8578fc16b7aae92739b82081a0e669d86afc36617d"}
Nov 03 09:15:15.528839 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:15.528789    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c826a7d-47qmk" podStartSLOduration=2.528774234 podStartE2EDuration="2.528774234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:15:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:15:15.526304534 +0000 UTC m=+2139.795367988" watchObservedRunningTime="2026-11-03 09:15:15.528774234 +0000 UTC m=+2139.797837667"
Nov 03 09:15:15.866220 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:15.866193    4473 scope.go:122] "RemoveContainer" containerID="3a89dec0928f515a50aa52518c6cfba66dab3274427d9c6f9026cb31fc1e34f6"
Nov 03 09:15:15.866367 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:15.866301    4473 scope.go:122] "RemoveContainer" containerID="7ab668affeebf7524d6a00f28621e135f613fc908f8e64f67b2ce46411d3b35b"
Nov 03 09:15:16.425724 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:16.425694    4473 generic.go:356] "Generic (PLEG): container finished" podID="750847c9-fd9f-4d39-8705-ccea1fd3fc17" containerID="d792438f2c04965b9ad4ffcbc69ac45726e4ed5cc8e95e7bd8b5e81dce1d44d8" exitCode=60
Nov 03 09:15:16.426197 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:16.425764    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894955-7v5xg" event={"ID":"750847c9-fd9f-4d39-8705-ccea1fd3fc17","Type":"ContainerDied","Data":"d792438f2c04965b9ad4ffcbc69ac45726e4ed5cc8e95e7bd8b5e81dce1d44d8"}
Nov 03 09:15:16.426197 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:16.425835    4473 scope.go:122] "RemoveContainer" containerID="7bdb2ee344359679c0d97b2a1cd6f52314c83a28252f54658b12879f412161b2"
Nov 03 09:15:16.426300 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:16.426224    4473 scope.go:122] "RemoveContainer" containerID="d792438f2c04965b9ad4ffcbc69ac45726e4ed5cc8e95e7bd8b5e81dce1d44d8"
Nov 03 09:15:16.426568 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:15:16.426549    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-77207e-29894955-7v5xg_odh-ai-gateway-infra(750847c9-fd9f-4d39-8705-ccea1fd3fc17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894955-7v5xg" podUID="750847c9-fd9f-4d39-8705-ccea1fd3fc17"
Nov 03 09:15:16.427878 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:16.427858    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894955-z25mc" event={"ID":"4cca539f-d6ea-4402-ac67-28886b17c397","Type":"ContainerStarted","Data":"512d99cb9fb41eb94f4ebb018fdd27da24b5df3387bc4b3bad65570cadd1d735"}
Nov 03 09:15:16.429709 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:16.429692    4473 generic.go:356] "Generic (PLEG): container finished" podID="00db03c4-9076-4e59-a799-a260f34a1e7d" containerID="189d5f1a2b5c31b54cf32b8578fc16b7aae92739b82081a0e669d86afc36617d" exitCode=60
Nov 03 09:15:16.429798 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:16.429758    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-121ee6-29894955-6nz68" event={"ID":"00db03c4-9076-4e59-a799-a260f34a1e7d","Type":"ContainerDied","Data":"189d5f1a2b5c31b54cf32b8578fc16b7aae92739b82081a0e669d86afc36617d"}
Nov 03 09:15:16.430062 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:16.430044    4473 scope.go:122] "RemoveContainer" containerID="189d5f1a2b5c31b54cf32b8578fc16b7aae92739b82081a0e669d86afc36617d"
Nov 03 09:15:16.430280 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:15:16.430257    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-121ee6-29894955-6nz68_odh-ai-gateway-infra(00db03c4-9076-4e59-a799-a260f34a1e7d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-121ee6-29894955-6nz68" podUID="00db03c4-9076-4e59-a799-a260f34a1e7d"
Nov 03 09:15:16.431428 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:16.431409    4473 generic.go:356] "Generic (PLEG): container finished" podID="17acd40d-9a11-4a0e-9a09-88f9756e2402" containerID="bb84b2efe161ca4b53b561e03a8d54f19042871f0233bbf499a0a692dc863043" exitCode=0
Nov 03 09:15:16.431534 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:16.431487    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c826a7d-47qmk" event={"ID":"17acd40d-9a11-4a0e-9a09-88f9756e2402","Type":"ContainerDied","Data":"bb84b2efe161ca4b53b561e03a8d54f19042871f0233bbf499a0a692dc863043"}
Nov 03 09:15:16.433333 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:16.433313    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894955-777bf" event={"ID":"20ec7053-908f-46d0-8b5e-f3fdb57d433e","Type":"ContainerStarted","Data":"188b89d7696b6235ac4a896c54a23e9bec7c5ef745a478e30ba7e9d95d670b7b"}
Nov 03 09:15:16.438804 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:16.438783    4473 scope.go:122] "RemoveContainer" containerID="f547a2110044b1bc62c502c29c594382c2122e6d86ed26e23ee3cc63dc742edb"
Nov 03 09:15:16.654741 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:16.654711    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894955-7v5xg"]
Nov 03 09:15:16.773111 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:16.773086    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-121ee6-29894955-6nz68"]
Nov 03 09:15:16.864986 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:16.863675    4473 scope.go:122] "RemoveContainer" containerID="c33b7894dcb2e44341fc6a1175d8bfb5cf5e1d95395d4036293d597865e24f7c"
Nov 03 09:15:16.937413 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:15:16.937380    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:15:17.309345 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:17.309265    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"]
Nov 03 09:15:17.309540 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:17.309480    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4" podUID="5571ff39-4a42-4390-aab3-2f1e60011a4e" containerName="istio-proxy" containerID="cri-o://e14b205c816fe103644b4fe18af95cd706fad49f77066b724dbd37700c997ff6" gracePeriod=30
Nov 03 09:15:17.476966 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:17.474849    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-133b31-29894955-wtst5" event={"ID":"bf19596b-0d54-4f5f-a6d9-433b4c130cbc","Type":"ContainerStarted","Data":"0588843a6f21c09c8fbf0c1b64a608f278ff8d34c5cb30390a36886fa18c1816"}
Nov 03 09:15:17.490967 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:17.490132    4473 generic.go:356] "Generic (PLEG): container finished" podID="20ec7053-908f-46d0-8b5e-f3fdb57d433e" containerID="188b89d7696b6235ac4a896c54a23e9bec7c5ef745a478e30ba7e9d95d670b7b" exitCode=60
Nov 03 09:15:17.490967 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:17.490294    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894955-777bf" event={"ID":"20ec7053-908f-46d0-8b5e-f3fdb57d433e","Type":"ContainerDied","Data":"188b89d7696b6235ac4a896c54a23e9bec7c5ef745a478e30ba7e9d95d670b7b"}
Nov 03 09:15:17.490967 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:17.490352    4473 scope.go:122] "RemoveContainer" containerID="3a89dec0928f515a50aa52518c6cfba66dab3274427d9c6f9026cb31fc1e34f6"
Nov 03 09:15:17.490967 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:17.490800    4473 scope.go:122] "RemoveContainer" containerID="188b89d7696b6235ac4a896c54a23e9bec7c5ef745a478e30ba7e9d95d670b7b"
Nov 03 09:15:17.491339 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:15:17.491175    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-26e861-29894955-777bf_odh-ai-gateway-infra(20ec7053-908f-46d0-8b5e-f3fdb57d433e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894955-777bf" podUID="20ec7053-908f-46d0-8b5e-f3fdb57d433e"
Nov 03 09:15:17.520308 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:17.517903    4473 generic.go:356] "Generic (PLEG): container finished" podID="4cca539f-d6ea-4402-ac67-28886b17c397" containerID="512d99cb9fb41eb94f4ebb018fdd27da24b5df3387bc4b3bad65570cadd1d735" exitCode=60
Nov 03 09:15:17.520308 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:17.518145    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894955-z25mc" event={"ID":"4cca539f-d6ea-4402-ac67-28886b17c397","Type":"ContainerDied","Data":"512d99cb9fb41eb94f4ebb018fdd27da24b5df3387bc4b3bad65570cadd1d735"}
Nov 03 09:15:17.520308 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:17.518600    4473 scope.go:122] "RemoveContainer" containerID="512d99cb9fb41eb94f4ebb018fdd27da24b5df3387bc4b3bad65570cadd1d735"
Nov 03 09:15:17.527056 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:15:17.525116    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-910d50-29894955-z25mc_odh-ai-gateway-infra(4cca539f-d6ea-4402-ac67-28886b17c397)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894955-z25mc" podUID="4cca539f-d6ea-4402-ac67-28886b17c397"
Nov 03 09:15:17.550034 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:17.546192    4473 scope.go:122] "RemoveContainer" containerID="7ab668affeebf7524d6a00f28621e135f613fc908f8e64f67b2ce46411d3b35b"
Nov 03 09:15:17.721980 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:17.721924    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894955-7v5xg"
Nov 03 09:15:17.785770 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:17.785746    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894955-777bf"]
Nov 03 09:15:17.789543 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:17.789520    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/750847c9-fd9f-4d39-8705-ccea1fd3fc17-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/750847c9-fd9f-4d39-8705-ccea1fd3fc17-maas-api-service-ca\") pod \"750847c9-fd9f-4d39-8705-ccea1fd3fc17\" (UID: \"750847c9-fd9f-4d39-8705-ccea1fd3fc17\") "
Nov 03 09:15:17.789667 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:17.789564    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/750847c9-fd9f-4d39-8705-ccea1fd3fc17-kube-api-access-4c5x5\" (UniqueName: \"kubernetes.io/projected/750847c9-fd9f-4d39-8705-ccea1fd3fc17-kube-api-access-4c5x5\") pod \"750847c9-fd9f-4d39-8705-ccea1fd3fc17\" (UID: \"750847c9-fd9f-4d39-8705-ccea1fd3fc17\") "
Nov 03 09:15:17.790093 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:17.789878    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/750847c9-fd9f-4d39-8705-ccea1fd3fc17-maas-api-service-ca" pod "750847c9-fd9f-4d39-8705-ccea1fd3fc17" (UID: "750847c9-fd9f-4d39-8705-ccea1fd3fc17"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:15:17.790253 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:17.790235    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/750847c9-fd9f-4d39-8705-ccea1fd3fc17-maas-api-service-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:17.790520 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:17.790498    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c826a7d-47qmk"
Nov 03 09:15:17.792384 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:17.792366    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/750847c9-fd9f-4d39-8705-ccea1fd3fc17-kube-api-access-4c5x5" pod "750847c9-fd9f-4d39-8705-ccea1fd3fc17" (UID: "750847c9-fd9f-4d39-8705-ccea1fd3fc17"). InnerVolumeSpecName "kube-api-access-4c5x5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:15:17.801130 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:17.801111    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-121ee6-29894955-6nz68"
Nov 03 09:15:17.863765 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:17.863691    4473 scope.go:122] "RemoveContainer" containerID="0f8d8cac4f1a99ee25b0aeac36db83fed94ef94302ae6a44180a55dfcfb7724f"
Nov 03 09:15:17.890490 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:17.890469    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17acd40d-9a11-4a0e-9a09-88f9756e2402-kube-api-access-vhp27\" (UniqueName: \"kubernetes.io/projected/17acd40d-9a11-4a0e-9a09-88f9756e2402-kube-api-access-vhp27\") pod \"17acd40d-9a11-4a0e-9a09-88f9756e2402\" (UID: \"17acd40d-9a11-4a0e-9a09-88f9756e2402\") "
Nov 03 09:15:17.890594 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:17.890522    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00db03c4-9076-4e59-a799-a260f34a1e7d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00db03c4-9076-4e59-a799-a260f34a1e7d-maas-api-service-ca\") pod \"00db03c4-9076-4e59-a799-a260f34a1e7d\" (UID: \"00db03c4-9076-4e59-a799-a260f34a1e7d\") "
Nov 03 09:15:17.890645 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:17.890600    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17acd40d-9a11-4a0e-9a09-88f9756e2402-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17acd40d-9a11-4a0e-9a09-88f9756e2402-maas-api-xks-ca\") pod \"17acd40d-9a11-4a0e-9a09-88f9756e2402\" (UID: \"17acd40d-9a11-4a0e-9a09-88f9756e2402\") "
Nov 03 09:15:17.890645 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:17.890632    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17acd40d-9a11-4a0e-9a09-88f9756e2402-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17acd40d-9a11-4a0e-9a09-88f9756e2402-maas-api-openshift-ca\") pod \"17acd40d-9a11-4a0e-9a09-88f9756e2402\" (UID: \"17acd40d-9a11-4a0e-9a09-88f9756e2402\") "
Nov 03 09:15:17.890728 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:17.890711    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00db03c4-9076-4e59-a799-a260f34a1e7d-kube-api-access-cf96s\" (UniqueName: \"kubernetes.io/projected/00db03c4-9076-4e59-a799-a260f34a1e7d-kube-api-access-cf96s\") pod \"00db03c4-9076-4e59-a799-a260f34a1e7d\" (UID: \"00db03c4-9076-4e59-a799-a260f34a1e7d\") "
Nov 03 09:15:17.891126 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:17.890862    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00db03c4-9076-4e59-a799-a260f34a1e7d-maas-api-service-ca" pod "00db03c4-9076-4e59-a799-a260f34a1e7d" (UID: "00db03c4-9076-4e59-a799-a260f34a1e7d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:15:17.891215 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:17.891147    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17acd40d-9a11-4a0e-9a09-88f9756e2402-maas-api-openshift-ca" pod "17acd40d-9a11-4a0e-9a09-88f9756e2402" (UID: "17acd40d-9a11-4a0e-9a09-88f9756e2402"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:15:17.891519 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:17.891491    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c5x5\" (UniqueName: \"kubernetes.io/projected/750847c9-fd9f-4d39-8705-ccea1fd3fc17-kube-api-access-4c5x5\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:17.891597 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:17.891524    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00db03c4-9076-4e59-a799-a260f34a1e7d-maas-api-service-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:17.891597 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:17.891539    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17acd40d-9a11-4a0e-9a09-88f9756e2402-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:17.893273 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:17.893248    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17acd40d-9a11-4a0e-9a09-88f9756e2402-maas-api-xks-ca" pod "17acd40d-9a11-4a0e-9a09-88f9756e2402" (UID: "17acd40d-9a11-4a0e-9a09-88f9756e2402"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:15:17.893360 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:17.893321    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17acd40d-9a11-4a0e-9a09-88f9756e2402-kube-api-access-vhp27" pod "17acd40d-9a11-4a0e-9a09-88f9756e2402" (UID: "17acd40d-9a11-4a0e-9a09-88f9756e2402"). InnerVolumeSpecName "kube-api-access-vhp27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:15:17.893416 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:17.893400    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00db03c4-9076-4e59-a799-a260f34a1e7d-kube-api-access-cf96s" pod "00db03c4-9076-4e59-a799-a260f34a1e7d" (UID: "00db03c4-9076-4e59-a799-a260f34a1e7d"). InnerVolumeSpecName "kube-api-access-cf96s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:15:17.992418 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:17.992381    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf96s\" (UniqueName: \"kubernetes.io/projected/00db03c4-9076-4e59-a799-a260f34a1e7d-kube-api-access-cf96s\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:17.992418 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:17.992411    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhp27\" (UniqueName: \"kubernetes.io/projected/17acd40d-9a11-4a0e-9a09-88f9756e2402-kube-api-access-vhp27\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:17.992418 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:17.992422    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17acd40d-9a11-4a0e-9a09-88f9756e2402-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:18.027736 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:18.027710    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894955-z25mc"]
Nov 03 09:15:18.537745 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:18.537659    4473 generic.go:356] "Generic (PLEG): container finished" podID="bf19596b-0d54-4f5f-a6d9-433b4c130cbc" containerID="0588843a6f21c09c8fbf0c1b64a608f278ff8d34c5cb30390a36886fa18c1816" exitCode=60
Nov 03 09:15:18.537745 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:18.537730    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-133b31-29894955-wtst5" event={"ID":"bf19596b-0d54-4f5f-a6d9-433b4c130cbc","Type":"ContainerDied","Data":"0588843a6f21c09c8fbf0c1b64a608f278ff8d34c5cb30390a36886fa18c1816"}
Nov 03 09:15:18.538295 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:18.537808    4473 scope.go:122] "RemoveContainer" containerID="c33b7894dcb2e44341fc6a1175d8bfb5cf5e1d95395d4036293d597865e24f7c"
Nov 03 09:15:18.538295 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:18.538276    4473 scope.go:122] "RemoveContainer" containerID="0588843a6f21c09c8fbf0c1b64a608f278ff8d34c5cb30390a36886fa18c1816"
Nov 03 09:15:18.538641 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:15:18.538615    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-133b31-29894955-wtst5_odh-ai-gateway-infra(bf19596b-0d54-4f5f-a6d9-433b4c130cbc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-133b31-29894955-wtst5" podUID="bf19596b-0d54-4f5f-a6d9-433b4c130cbc"
Nov 03 09:15:18.540939 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:18.540914    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894955-7v5xg"
Nov 03 09:15:18.540939 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:18.540926    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894955-7v5xg" event={"ID":"750847c9-fd9f-4d39-8705-ccea1fd3fc17","Type":"ContainerDied","Data":"8ab98ce00b9a9b22f271e850dee6ec5222c515c1ce414bb85ffb93b3acef8af8"}
Nov 03 09:15:18.545986 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:18.545922    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-121ee6-29894955-6nz68" event={"ID":"00db03c4-9076-4e59-a799-a260f34a1e7d","Type":"ContainerDied","Data":"8ca8be60c5639ee27c7eeeb98ac760683b50a94319de78e73c422094c58ecd04"}
Nov 03 09:15:18.546097 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:18.546032    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-121ee6-29894955-6nz68"
Nov 03 09:15:18.558167 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:18.558142    4473 scope.go:122] "RemoveContainer" containerID="d792438f2c04965b9ad4ffcbc69ac45726e4ed5cc8e95e7bd8b5e81dce1d44d8"
Nov 03 09:15:18.559103 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:18.559076    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c826a7d-47qmk"
Nov 03 09:15:18.559221 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:18.559074    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c826a7d-47qmk" event={"ID":"17acd40d-9a11-4a0e-9a09-88f9756e2402","Type":"ContainerDied","Data":"bc7bec9bda92132dd7b157202c929fd39c388f0c008e3b3c4cf35c9d7622d0d4"}
Nov 03 09:15:18.559221 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:18.559207    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc7bec9bda92132dd7b157202c929fd39c388f0c008e3b3c4cf35c9d7622d0d4"
Nov 03 09:15:18.561528 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:18.561502    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894955-58wlg" event={"ID":"61f3f0ff-1cbf-4e08-8d44-f44952e3b01c","Type":"ContainerStarted","Data":"876e2671479f0fb168cf8eee04bb82135a34ac3b0f09d099877436e8f28c41ae"}
Nov 03 09:15:18.570402 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:18.570297    4473 scope.go:122] "RemoveContainer" containerID="189d5f1a2b5c31b54cf32b8578fc16b7aae92739b82081a0e669d86afc36617d"
Nov 03 09:15:18.630679 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:18.628864    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-133b31-29894955-wtst5"]
Nov 03 09:15:18.705401 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:18.705374    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894955-777bf"
Nov 03 09:15:18.737555 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:18.737520    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894955-z25mc"
Nov 03 09:15:18.802221 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:18.802156    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20ec7053-908f-46d0-8b5e-f3fdb57d433e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20ec7053-908f-46d0-8b5e-f3fdb57d433e-maas-api-service-ca\") pod \"20ec7053-908f-46d0-8b5e-f3fdb57d433e\" (UID: \"20ec7053-908f-46d0-8b5e-f3fdb57d433e\") "
Nov 03 09:15:18.802355 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:18.802257    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cca539f-d6ea-4402-ac67-28886b17c397-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cca539f-d6ea-4402-ac67-28886b17c397-maas-api-service-ca\") pod \"4cca539f-d6ea-4402-ac67-28886b17c397\" (UID: \"4cca539f-d6ea-4402-ac67-28886b17c397\") "
Nov 03 09:15:18.802355 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:18.802302    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cca539f-d6ea-4402-ac67-28886b17c397-kube-api-access-8k8z2\" (UniqueName: \"kubernetes.io/projected/4cca539f-d6ea-4402-ac67-28886b17c397-kube-api-access-8k8z2\") pod \"4cca539f-d6ea-4402-ac67-28886b17c397\" (UID: \"4cca539f-d6ea-4402-ac67-28886b17c397\") "
Nov 03 09:15:18.802355 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:18.802323    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20ec7053-908f-46d0-8b5e-f3fdb57d433e-kube-api-access-tfrw8\" (UniqueName: \"kubernetes.io/projected/20ec7053-908f-46d0-8b5e-f3fdb57d433e-kube-api-access-tfrw8\") pod \"20ec7053-908f-46d0-8b5e-f3fdb57d433e\" (UID: \"20ec7053-908f-46d0-8b5e-f3fdb57d433e\") "
Nov 03 09:15:18.802561 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:18.802536    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20ec7053-908f-46d0-8b5e-f3fdb57d433e-maas-api-service-ca" pod "20ec7053-908f-46d0-8b5e-f3fdb57d433e" (UID: "20ec7053-908f-46d0-8b5e-f3fdb57d433e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:15:18.802607 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:18.802572    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cca539f-d6ea-4402-ac67-28886b17c397-maas-api-service-ca" pod "4cca539f-d6ea-4402-ac67-28886b17c397" (UID: "4cca539f-d6ea-4402-ac67-28886b17c397"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:15:18.802809 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:18.802789    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20ec7053-908f-46d0-8b5e-f3fdb57d433e-maas-api-service-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:18.803023 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:18.802812    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cca539f-d6ea-4402-ac67-28886b17c397-maas-api-service-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:18.804563 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:18.804545    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cca539f-d6ea-4402-ac67-28886b17c397-kube-api-access-8k8z2" pod "4cca539f-d6ea-4402-ac67-28886b17c397" (UID: "4cca539f-d6ea-4402-ac67-28886b17c397"). InnerVolumeSpecName "kube-api-access-8k8z2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:15:18.804563 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:18.804572    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20ec7053-908f-46d0-8b5e-f3fdb57d433e-kube-api-access-tfrw8" pod "20ec7053-908f-46d0-8b5e-f3fdb57d433e" (UID: "20ec7053-908f-46d0-8b5e-f3fdb57d433e"). InnerVolumeSpecName "kube-api-access-tfrw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:15:18.858527 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:18.858496    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894955-7v5xg"]
Nov 03 09:15:18.864464 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:18.864435    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894955-7v5xg"]
Nov 03 09:15:18.903809 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:18.903773    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k8z2\" (UniqueName: \"kubernetes.io/projected/4cca539f-d6ea-4402-ac67-28886b17c397-kube-api-access-8k8z2\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:18.903809 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:18.903797    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfrw8\" (UniqueName: \"kubernetes.io/projected/20ec7053-908f-46d0-8b5e-f3fdb57d433e-kube-api-access-tfrw8\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:19.093856 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:19.093775    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-121ee6-29894955-6nz68"]
Nov 03 09:15:19.103799 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:19.103759    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-121ee6-29894955-6nz68"]
Nov 03 09:15:19.573726 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:19.573696    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894955-777bf" event={"ID":"20ec7053-908f-46d0-8b5e-f3fdb57d433e","Type":"ContainerDied","Data":"52d8cc0e7ab436824c54a220394ae05bc23c9b9a1945d7923f063bdae30c57a1"}
Nov 03 09:15:19.573726 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:19.573720    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894955-777bf"
Nov 03 09:15:19.574225 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:19.573759    4473 scope.go:122] "RemoveContainer" containerID="188b89d7696b6235ac4a896c54a23e9bec7c5ef745a478e30ba7e9d95d670b7b"
Nov 03 09:15:19.577145 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:19.577124    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894955-z25mc"
Nov 03 09:15:19.577525 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:19.577121    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894955-z25mc" event={"ID":"4cca539f-d6ea-4402-ac67-28886b17c397","Type":"ContainerDied","Data":"d7a68b843cb633d5125e791691df66666b8d2d9c205d67288e682275643817b6"}
Nov 03 09:15:19.580973 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:19.580928    4473 generic.go:356] "Generic (PLEG): container finished" podID="61f3f0ff-1cbf-4e08-8d44-f44952e3b01c" containerID="876e2671479f0fb168cf8eee04bb82135a34ac3b0f09d099877436e8f28c41ae" exitCode=60
Nov 03 09:15:19.581087 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:19.580993    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894955-58wlg" event={"ID":"61f3f0ff-1cbf-4e08-8d44-f44952e3b01c","Type":"ContainerDied","Data":"876e2671479f0fb168cf8eee04bb82135a34ac3b0f09d099877436e8f28c41ae"}
Nov 03 09:15:19.581399 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:19.581381    4473 scope.go:122] "RemoveContainer" containerID="876e2671479f0fb168cf8eee04bb82135a34ac3b0f09d099877436e8f28c41ae"
Nov 03 09:15:19.581686 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:15:19.581662    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-18f9da-29894955-58wlg_odh-ai-gateway-infra(61f3f0ff-1cbf-4e08-8d44-f44952e3b01c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894955-58wlg" podUID="61f3f0ff-1cbf-4e08-8d44-f44952e3b01c"
Nov 03 09:15:19.591470 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:19.591253    4473 scope.go:122] "RemoveContainer" containerID="512d99cb9fb41eb94f4ebb018fdd27da24b5df3387bc4b3bad65570cadd1d735"
Nov 03 09:15:19.602920 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:19.602902    4473 scope.go:122] "RemoveContainer" containerID="0f8d8cac4f1a99ee25b0aeac36db83fed94ef94302ae6a44180a55dfcfb7724f"
Nov 03 09:15:19.754635 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:19.754612    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-133b31-29894955-wtst5"
Nov 03 09:15:19.802380 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:19.802353    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894955-z25mc"]
Nov 03 09:15:19.807277 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:19.807254    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894955-z25mc"]
Nov 03 09:15:19.814518 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:19.814494    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf19596b-0d54-4f5f-a6d9-433b4c130cbc-kube-api-access-kxdsx\" (UniqueName: \"kubernetes.io/projected/bf19596b-0d54-4f5f-a6d9-433b4c130cbc-kube-api-access-kxdsx\") pod \"bf19596b-0d54-4f5f-a6d9-433b4c130cbc\" (UID: \"bf19596b-0d54-4f5f-a6d9-433b4c130cbc\") "
Nov 03 09:15:19.814910 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:19.814559    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf19596b-0d54-4f5f-a6d9-433b4c130cbc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf19596b-0d54-4f5f-a6d9-433b4c130cbc-maas-api-service-ca\") pod \"bf19596b-0d54-4f5f-a6d9-433b4c130cbc\" (UID: \"bf19596b-0d54-4f5f-a6d9-433b4c130cbc\") "
Nov 03 09:15:19.815071 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:19.814983    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf19596b-0d54-4f5f-a6d9-433b4c130cbc-maas-api-service-ca" pod "bf19596b-0d54-4f5f-a6d9-433b4c130cbc" (UID: "bf19596b-0d54-4f5f-a6d9-433b4c130cbc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:15:19.816750 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:19.816727    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf19596b-0d54-4f5f-a6d9-433b4c130cbc-kube-api-access-kxdsx" pod "bf19596b-0d54-4f5f-a6d9-433b4c130cbc" (UID: "bf19596b-0d54-4f5f-a6d9-433b4c130cbc"). InnerVolumeSpecName "kube-api-access-kxdsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:15:19.870255 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:19.870224    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00db03c4-9076-4e59-a799-a260f34a1e7d" path="/var/lib/kubelet/pods/00db03c4-9076-4e59-a799-a260f34a1e7d/volumes"
Nov 03 09:15:19.870646 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:19.870632    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cca539f-d6ea-4402-ac67-28886b17c397" path="/var/lib/kubelet/pods/4cca539f-d6ea-4402-ac67-28886b17c397/volumes"
Nov 03 09:15:19.871018 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:19.871004    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="750847c9-fd9f-4d39-8705-ccea1fd3fc17" path="/var/lib/kubelet/pods/750847c9-fd9f-4d39-8705-ccea1fd3fc17/volumes"
Nov 03 09:15:19.916332 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:19.916301    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxdsx\" (UniqueName: \"kubernetes.io/projected/bf19596b-0d54-4f5f-a6d9-433b4c130cbc-kube-api-access-kxdsx\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:19.916332 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:19.916326    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf19596b-0d54-4f5f-a6d9-433b4c130cbc-maas-api-service-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:20.034718 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:20.034687    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894955-777bf"]
Nov 03 09:15:20.037450 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:20.037430    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-26e861-29894955-777bf"]
Nov 03 09:15:20.342401 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:20.342365    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894955-58wlg"]
Nov 03 09:15:20.589452 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:20.589421    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-133b31-29894955-wtst5" event={"ID":"bf19596b-0d54-4f5f-a6d9-433b4c130cbc","Type":"ContainerDied","Data":"912d37f877299c2b62af69417e05f1bb0409807a263066e948b8fcf432e2e7bd"}
Nov 03 09:15:20.589827 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:20.589430    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-133b31-29894955-wtst5"
Nov 03 09:15:20.589827 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:20.589479    4473 scope.go:122] "RemoveContainer" containerID="0588843a6f21c09c8fbf0c1b64a608f278ff8d34c5cb30390a36886fa18c1816"
Nov 03 09:15:20.741938 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:20.741900    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894955-58wlg"
Nov 03 09:15:20.822076 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:20.822034    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-133b31-29894955-wtst5"]
Nov 03 09:15:20.825509 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:20.825485    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61f3f0ff-1cbf-4e08-8d44-f44952e3b01c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61f3f0ff-1cbf-4e08-8d44-f44952e3b01c-maas-api-service-ca\") pod \"61f3f0ff-1cbf-4e08-8d44-f44952e3b01c\" (UID: \"61f3f0ff-1cbf-4e08-8d44-f44952e3b01c\") "
Nov 03 09:15:20.825642 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:20.825571    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61f3f0ff-1cbf-4e08-8d44-f44952e3b01c-kube-api-access-ctpl2\" (UniqueName: \"kubernetes.io/projected/61f3f0ff-1cbf-4e08-8d44-f44952e3b01c-kube-api-access-ctpl2\") pod \"61f3f0ff-1cbf-4e08-8d44-f44952e3b01c\" (UID: \"61f3f0ff-1cbf-4e08-8d44-f44952e3b01c\") "
Nov 03 09:15:20.825887 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:20.825864    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61f3f0ff-1cbf-4e08-8d44-f44952e3b01c-maas-api-service-ca" pod "61f3f0ff-1cbf-4e08-8d44-f44952e3b01c" (UID: "61f3f0ff-1cbf-4e08-8d44-f44952e3b01c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:15:20.826039 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:20.826003    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61f3f0ff-1cbf-4e08-8d44-f44952e3b01c-maas-api-service-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:20.827199 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:20.827167    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-133b31-29894955-wtst5"]
Nov 03 09:15:20.827822 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:20.827795    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f3f0ff-1cbf-4e08-8d44-f44952e3b01c-kube-api-access-ctpl2" pod "61f3f0ff-1cbf-4e08-8d44-f44952e3b01c" (UID: "61f3f0ff-1cbf-4e08-8d44-f44952e3b01c"). InnerVolumeSpecName "kube-api-access-ctpl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:15:20.927021 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:20.926980    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctpl2\" (UniqueName: \"kubernetes.io/projected/61f3f0ff-1cbf-4e08-8d44-f44952e3b01c-kube-api-access-ctpl2\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:21.599575 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:21.599541    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894955-58wlg"
Nov 03 09:15:21.599575 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:21.599554    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894955-58wlg" event={"ID":"61f3f0ff-1cbf-4e08-8d44-f44952e3b01c","Type":"ContainerDied","Data":"128f0c019279a21961e7fd4ded531a7477ac3ec123be16f28641f0e3235e2243"}
Nov 03 09:15:21.600148 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:21.599650    4473 scope.go:122] "RemoveContainer" containerID="876e2671479f0fb168cf8eee04bb82135a34ac3b0f09d099877436e8f28c41ae"
Nov 03 09:15:21.770848 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:21.770819    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894955-58wlg"]
Nov 03 09:15:21.776615 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:21.776590    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894955-58wlg"]
Nov 03 09:15:21.869384 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:21.869364    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20ec7053-908f-46d0-8b5e-f3fdb57d433e" path="/var/lib/kubelet/pods/20ec7053-908f-46d0-8b5e-f3fdb57d433e/volumes"
Nov 03 09:15:21.869836 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:21.869809    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61f3f0ff-1cbf-4e08-8d44-f44952e3b01c" path="/var/lib/kubelet/pods/61f3f0ff-1cbf-4e08-8d44-f44952e3b01c/volumes"
Nov 03 09:15:21.870237 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:21.870224    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf19596b-0d54-4f5f-a6d9-433b4c130cbc" path="/var/lib/kubelet/pods/bf19596b-0d54-4f5f-a6d9-433b4c130cbc/volumes"
Nov 03 09:15:22.575625 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:22.575602    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"
Nov 03 09:15:22.609245 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:22.609183    4473 generic.go:356] "Generic (PLEG): container finished" podID="5571ff39-4a42-4390-aab3-2f1e60011a4e" containerID="e14b205c816fe103644b4fe18af95cd706fad49f77066b724dbd37700c997ff6" exitCode=0
Nov 03 09:15:22.609245 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:22.609234    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"
Nov 03 09:15:22.609633 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:22.609269    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4" event={"ID":"5571ff39-4a42-4390-aab3-2f1e60011a4e","Type":"ContainerDied","Data":"e14b205c816fe103644b4fe18af95cd706fad49f77066b724dbd37700c997ff6"}
Nov 03 09:15:22.609633 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:22.609325    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4" event={"ID":"5571ff39-4a42-4390-aab3-2f1e60011a4e","Type":"ContainerDied","Data":"3e84375968fd17649647cf70da9acd7424d0467f6fc2682a6abb22639ae9dba1"}
Nov 03 09:15:22.609633 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:22.609350    4473 scope.go:122] "RemoveContainer" containerID="e14b205c816fe103644b4fe18af95cd706fad49f77066b724dbd37700c997ff6"
Nov 03 09:15:22.618394 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:22.618378    4473 scope.go:122] "RemoveContainer" containerID="f7812c462d79f52797cef5c89402e0d737854f743ce110893ab312c840d0e0b1"
Nov 03 09:15:22.626932 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:22.626909    4473 scope.go:122] "RemoveContainer" containerID="e14b205c816fe103644b4fe18af95cd706fad49f77066b724dbd37700c997ff6"
Nov 03 09:15:22.627218 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:15:22.627198    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e14b205c816fe103644b4fe18af95cd706fad49f77066b724dbd37700c997ff6\": container with ID starting with e14b205c816fe103644b4fe18af95cd706fad49f77066b724dbd37700c997ff6 not found: ID does not exist" containerID="e14b205c816fe103644b4fe18af95cd706fad49f77066b724dbd37700c997ff6"
Nov 03 09:15:22.627266 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:22.627226    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e14b205c816fe103644b4fe18af95cd706fad49f77066b724dbd37700c997ff6"} err="failed to get container status \"e14b205c816fe103644b4fe18af95cd706fad49f77066b724dbd37700c997ff6\": rpc error: code = NotFound desc = could not find container \"e14b205c816fe103644b4fe18af95cd706fad49f77066b724dbd37700c997ff6\": container with ID starting with e14b205c816fe103644b4fe18af95cd706fad49f77066b724dbd37700c997ff6 not found: ID does not exist"
Nov 03 09:15:22.627266 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:22.627242    4473 scope.go:122] "RemoveContainer" containerID="f7812c462d79f52797cef5c89402e0d737854f743ce110893ab312c840d0e0b1"
Nov 03 09:15:22.627484 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:15:22.627468    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7812c462d79f52797cef5c89402e0d737854f743ce110893ab312c840d0e0b1\": container with ID starting with f7812c462d79f52797cef5c89402e0d737854f743ce110893ab312c840d0e0b1 not found: ID does not exist" containerID="f7812c462d79f52797cef5c89402e0d737854f743ce110893ab312c840d0e0b1"
Nov 03 09:15:22.627526 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:22.627491    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7812c462d79f52797cef5c89402e0d737854f743ce110893ab312c840d0e0b1"} err="failed to get container status \"f7812c462d79f52797cef5c89402e0d737854f743ce110893ab312c840d0e0b1\": rpc error: code = NotFound desc = could not find container \"f7812c462d79f52797cef5c89402e0d737854f743ce110893ab312c840d0e0b1\": container with ID starting with f7812c462d79f52797cef5c89402e0d737854f743ce110893ab312c840d0e0b1 not found: ID does not exist"
Nov 03 09:15:22.646470 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:22.646451    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5571ff39-4a42-4390-aab3-2f1e60011a4e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5571ff39-4a42-4390-aab3-2f1e60011a4e-workload-socket\") pod \"5571ff39-4a42-4390-aab3-2f1e60011a4e\" (UID: \"5571ff39-4a42-4390-aab3-2f1e60011a4e\") "
Nov 03 09:15:22.646551 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:22.646483    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5571ff39-4a42-4390-aab3-2f1e60011a4e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5571ff39-4a42-4390-aab3-2f1e60011a4e-workload-certs\") pod \"5571ff39-4a42-4390-aab3-2f1e60011a4e\" (UID: \"5571ff39-4a42-4390-aab3-2f1e60011a4e\") "
Nov 03 09:15:22.646551 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:22.646510    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5571ff39-4a42-4390-aab3-2f1e60011a4e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5571ff39-4a42-4390-aab3-2f1e60011a4e-istio-envoy\") pod \"5571ff39-4a42-4390-aab3-2f1e60011a4e\" (UID: \"5571ff39-4a42-4390-aab3-2f1e60011a4e\") "
Nov 03 09:15:22.646655 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:22.646589    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5571ff39-4a42-4390-aab3-2f1e60011a4e-istio-token\" (UniqueName: \"kubernetes.io/projected/5571ff39-4a42-4390-aab3-2f1e60011a4e-istio-token\") pod \"5571ff39-4a42-4390-aab3-2f1e60011a4e\" (UID: \"5571ff39-4a42-4390-aab3-2f1e60011a4e\") "
Nov 03 09:15:22.646655 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:22.646626    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5571ff39-4a42-4390-aab3-2f1e60011a4e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5571ff39-4a42-4390-aab3-2f1e60011a4e-istiod-ca-cert\") pod \"5571ff39-4a42-4390-aab3-2f1e60011a4e\" (UID: \"5571ff39-4a42-4390-aab3-2f1e60011a4e\") "
Nov 03 09:15:22.646655 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:22.646649    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5571ff39-4a42-4390-aab3-2f1e60011a4e-kube-api-access-b2896\" (UniqueName: \"kubernetes.io/projected/5571ff39-4a42-4390-aab3-2f1e60011a4e-kube-api-access-b2896\") pod \"5571ff39-4a42-4390-aab3-2f1e60011a4e\" (UID: \"5571ff39-4a42-4390-aab3-2f1e60011a4e\") "
Nov 03 09:15:22.646808 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:22.646675    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5571ff39-4a42-4390-aab3-2f1e60011a4e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5571ff39-4a42-4390-aab3-2f1e60011a4e-credential-socket\") pod \"5571ff39-4a42-4390-aab3-2f1e60011a4e\" (UID: \"5571ff39-4a42-4390-aab3-2f1e60011a4e\") "
Nov 03 09:15:22.646808 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:22.646694    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5571ff39-4a42-4390-aab3-2f1e60011a4e-workload-socket" pod "5571ff39-4a42-4390-aab3-2f1e60011a4e" (UID: "5571ff39-4a42-4390-aab3-2f1e60011a4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:15:22.646808 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:22.646739    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5571ff39-4a42-4390-aab3-2f1e60011a4e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5571ff39-4a42-4390-aab3-2f1e60011a4e-istio-data\") pod \"5571ff39-4a42-4390-aab3-2f1e60011a4e\" (UID: \"5571ff39-4a42-4390-aab3-2f1e60011a4e\") "
Nov 03 09:15:22.646808 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:22.646741    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5571ff39-4a42-4390-aab3-2f1e60011a4e-workload-certs" pod "5571ff39-4a42-4390-aab3-2f1e60011a4e" (UID: "5571ff39-4a42-4390-aab3-2f1e60011a4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:15:22.646808 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:22.646787    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5571ff39-4a42-4390-aab3-2f1e60011a4e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5571ff39-4a42-4390-aab3-2f1e60011a4e-istio-podinfo\") pod \"5571ff39-4a42-4390-aab3-2f1e60011a4e\" (UID: \"5571ff39-4a42-4390-aab3-2f1e60011a4e\") "
Nov 03 09:15:22.647097 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:22.647014    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5571ff39-4a42-4390-aab3-2f1e60011a4e-istiod-ca-cert" pod "5571ff39-4a42-4390-aab3-2f1e60011a4e" (UID: "5571ff39-4a42-4390-aab3-2f1e60011a4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:15:22.647157 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:22.647099    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5571ff39-4a42-4390-aab3-2f1e60011a4e-istio-data" pod "5571ff39-4a42-4390-aab3-2f1e60011a4e" (UID: "5571ff39-4a42-4390-aab3-2f1e60011a4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:15:22.647542 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:22.647508    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5571ff39-4a42-4390-aab3-2f1e60011a4e-credential-socket" pod "5571ff39-4a42-4390-aab3-2f1e60011a4e" (UID: "5571ff39-4a42-4390-aab3-2f1e60011a4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:15:22.647542 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:22.647526    4473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5571ff39-4a42-4390-aab3-2f1e60011a4e-workload-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:22.647698 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:22.647550    4473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5571ff39-4a42-4390-aab3-2f1e60011a4e-workload-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:22.647698 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:22.647566    4473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5571ff39-4a42-4390-aab3-2f1e60011a4e-istiod-ca-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:22.647698 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:22.647580    4473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5571ff39-4a42-4390-aab3-2f1e60011a4e-istio-data\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:22.649072 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:22.649048    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5571ff39-4a42-4390-aab3-2f1e60011a4e-kube-api-access-b2896" pod "5571ff39-4a42-4390-aab3-2f1e60011a4e" (UID: "5571ff39-4a42-4390-aab3-2f1e60011a4e"). InnerVolumeSpecName "kube-api-access-b2896". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:15:22.649160 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:22.649096    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5571ff39-4a42-4390-aab3-2f1e60011a4e-istio-token" pod "5571ff39-4a42-4390-aab3-2f1e60011a4e" (UID: "5571ff39-4a42-4390-aab3-2f1e60011a4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:15:22.649160 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:22.649128    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5571ff39-4a42-4390-aab3-2f1e60011a4e-istio-envoy" pod "5571ff39-4a42-4390-aab3-2f1e60011a4e" (UID: "5571ff39-4a42-4390-aab3-2f1e60011a4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:15:22.649265 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:22.649248    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5571ff39-4a42-4390-aab3-2f1e60011a4e-istio-podinfo" pod "5571ff39-4a42-4390-aab3-2f1e60011a4e" (UID: "5571ff39-4a42-4390-aab3-2f1e60011a4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 09:15:22.749164 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:22.749136    4473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5571ff39-4a42-4390-aab3-2f1e60011a4e-istio-token\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:22.749318 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:22.749169    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2896\" (UniqueName: \"kubernetes.io/projected/5571ff39-4a42-4390-aab3-2f1e60011a4e-kube-api-access-b2896\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:22.749318 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:22.749185    4473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5571ff39-4a42-4390-aab3-2f1e60011a4e-credential-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:22.749318 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:22.749198    4473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5571ff39-4a42-4390-aab3-2f1e60011a4e-istio-podinfo\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:22.749318 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:22.749210    4473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5571ff39-4a42-4390-aab3-2f1e60011a4e-istio-envoy\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:23.035763 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:23.035732    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"]
Nov 03 09:15:23.042173 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:23.042141    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-04934e-openshift-default-7496c6d445-hs9d4"]
Nov 03 09:15:23.869497 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:23.869468    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5571ff39-4a42-4390-aab3-2f1e60011a4e" path="/var/lib/kubelet/pods/5571ff39-4a42-4390-aab3-2f1e60011a4e/volumes"
Nov 03 09:15:24.781724 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.781683    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b2d37a3-zglmc"]
Nov 03 09:15:24.782489 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.782463    4473 cpu_manager.go:414] "containerMap: removing container" podUID="00db03c4-9076-4e59-a799-a260f34a1e7d" containerName="cleanup"
Nov 03 09:15:24.782489 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.782489    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00db03c4-9076-4e59-a799-a260f34a1e7d" containerName="cleanup"
Nov 03 09:15:24.782691 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.782512    4473 cpu_manager.go:414] "containerMap: removing container" podUID="750847c9-fd9f-4d39-8705-ccea1fd3fc17" containerName="cleanup"
Nov 03 09:15:24.782691 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.782521    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="750847c9-fd9f-4d39-8705-ccea1fd3fc17" containerName="cleanup"
Nov 03 09:15:24.782691 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.782538    4473 cpu_manager.go:414] "containerMap: removing container" podUID="750847c9-fd9f-4d39-8705-ccea1fd3fc17" containerName="cleanup"
Nov 03 09:15:24.782691 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.782546    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="750847c9-fd9f-4d39-8705-ccea1fd3fc17" containerName="cleanup"
Nov 03 09:15:24.782691 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.782564    4473 cpu_manager.go:414] "containerMap: removing container" podUID="00db03c4-9076-4e59-a799-a260f34a1e7d" containerName="cleanup"
Nov 03 09:15:24.782691 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.782570    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00db03c4-9076-4e59-a799-a260f34a1e7d" containerName="cleanup"
Nov 03 09:15:24.782691 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.782589    4473 cpu_manager.go:414] "containerMap: removing container" podUID="4cca539f-d6ea-4402-ac67-28886b17c397" containerName="cleanup"
Nov 03 09:15:24.782691 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.782596    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cca539f-d6ea-4402-ac67-28886b17c397" containerName="cleanup"
Nov 03 09:15:24.782691 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.782605    4473 cpu_manager.go:414] "containerMap: removing container" podUID="20ec7053-908f-46d0-8b5e-f3fdb57d433e" containerName="cleanup"
Nov 03 09:15:24.782691 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.782612    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20ec7053-908f-46d0-8b5e-f3fdb57d433e" containerName="cleanup"
Nov 03 09:15:24.782691 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.782622    4473 cpu_manager.go:414] "containerMap: removing container" podUID="bf19596b-0d54-4f5f-a6d9-433b4c130cbc" containerName="cleanup"
Nov 03 09:15:24.782691 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.782630    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf19596b-0d54-4f5f-a6d9-433b4c130cbc" containerName="cleanup"
Nov 03 09:15:24.782691 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.782641    4473 cpu_manager.go:414] "containerMap: removing container" podUID="61f3f0ff-1cbf-4e08-8d44-f44952e3b01c" containerName="cleanup"
Nov 03 09:15:24.782691 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.782647    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61f3f0ff-1cbf-4e08-8d44-f44952e3b01c" containerName="cleanup"
Nov 03 09:15:24.782691 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.782657    4473 cpu_manager.go:414] "containerMap: removing container" podUID="17acd40d-9a11-4a0e-9a09-88f9756e2402" containerName="revoke-keys"
Nov 03 09:15:24.782691 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.782663    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17acd40d-9a11-4a0e-9a09-88f9756e2402" containerName="revoke-keys"
Nov 03 09:15:24.782691 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.782675    4473 cpu_manager.go:414] "containerMap: removing container" podUID="4cca539f-d6ea-4402-ac67-28886b17c397" containerName="cleanup"
Nov 03 09:15:24.782691 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.782682    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cca539f-d6ea-4402-ac67-28886b17c397" containerName="cleanup"
Nov 03 09:15:24.782691 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.782690    4473 cpu_manager.go:414] "containerMap: removing container" podUID="5571ff39-4a42-4390-aab3-2f1e60011a4e" containerName="istio-proxy"
Nov 03 09:15:24.782691 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.782699    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5571ff39-4a42-4390-aab3-2f1e60011a4e" containerName="istio-proxy"
Nov 03 09:15:24.783749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.782709    4473 cpu_manager.go:414] "containerMap: removing container" podUID="4cca539f-d6ea-4402-ac67-28886b17c397" containerName="cleanup"
Nov 03 09:15:24.783749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.782716    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cca539f-d6ea-4402-ac67-28886b17c397" containerName="cleanup"
Nov 03 09:15:24.783749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.782745    4473 cpu_manager.go:414] "containerMap: removing container" podUID="20ec7053-908f-46d0-8b5e-f3fdb57d433e" containerName="cleanup"
Nov 03 09:15:24.783749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.782752    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20ec7053-908f-46d0-8b5e-f3fdb57d433e" containerName="cleanup"
Nov 03 09:15:24.783749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.782762    4473 cpu_manager.go:414] "containerMap: removing container" podUID="bf19596b-0d54-4f5f-a6d9-433b4c130cbc" containerName="cleanup"
Nov 03 09:15:24.783749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.782770    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf19596b-0d54-4f5f-a6d9-433b4c130cbc" containerName="cleanup"
Nov 03 09:15:24.783749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.782788    4473 cpu_manager.go:414] "containerMap: removing container" podUID="61f3f0ff-1cbf-4e08-8d44-f44952e3b01c" containerName="cleanup"
Nov 03 09:15:24.783749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.782796    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61f3f0ff-1cbf-4e08-8d44-f44952e3b01c" containerName="cleanup"
Nov 03 09:15:24.783749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.782815    4473 cpu_manager.go:414] "containerMap: removing container" podUID="20ec7053-908f-46d0-8b5e-f3fdb57d433e" containerName="cleanup"
Nov 03 09:15:24.783749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.782822    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20ec7053-908f-46d0-8b5e-f3fdb57d433e" containerName="cleanup"
Nov 03 09:15:24.783749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.782831    4473 cpu_manager.go:414] "containerMap: removing container" podUID="61f3f0ff-1cbf-4e08-8d44-f44952e3b01c" containerName="cleanup"
Nov 03 09:15:24.783749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.782838    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61f3f0ff-1cbf-4e08-8d44-f44952e3b01c" containerName="cleanup"
Nov 03 09:15:24.783749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.782854    4473 cpu_manager.go:414] "containerMap: removing container" podUID="5571ff39-4a42-4390-aab3-2f1e60011a4e" containerName="istio-proxy"
Nov 03 09:15:24.783749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.782861    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5571ff39-4a42-4390-aab3-2f1e60011a4e" containerName="istio-proxy"
Nov 03 09:15:24.783749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.783026    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cca539f-d6ea-4402-ac67-28886b17c397" containerName="cleanup"
Nov 03 09:15:24.783749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.783040    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="20ec7053-908f-46d0-8b5e-f3fdb57d433e" containerName="cleanup"
Nov 03 09:15:24.783749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.783050    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf19596b-0d54-4f5f-a6d9-433b4c130cbc" containerName="cleanup"
Nov 03 09:15:24.783749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.783060    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="61f3f0ff-1cbf-4e08-8d44-f44952e3b01c" containerName="cleanup"
Nov 03 09:15:24.783749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.783067    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="17acd40d-9a11-4a0e-9a09-88f9756e2402" containerName="revoke-keys"
Nov 03 09:15:24.783749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.783074    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cca539f-d6ea-4402-ac67-28886b17c397" containerName="cleanup"
Nov 03 09:15:24.783749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.783084    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="5571ff39-4a42-4390-aab3-2f1e60011a4e" containerName="istio-proxy"
Nov 03 09:15:24.783749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.783092    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cca539f-d6ea-4402-ac67-28886b17c397" containerName="cleanup"
Nov 03 09:15:24.783749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.783104    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="750847c9-fd9f-4d39-8705-ccea1fd3fc17" containerName="cleanup"
Nov 03 09:15:24.783749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.783113    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="20ec7053-908f-46d0-8b5e-f3fdb57d433e" containerName="cleanup"
Nov 03 09:15:24.783749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.783122    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf19596b-0d54-4f5f-a6d9-433b4c130cbc" containerName="cleanup"
Nov 03 09:15:24.783749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.783129    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="00db03c4-9076-4e59-a799-a260f34a1e7d" containerName="cleanup"
Nov 03 09:15:24.783749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.783141    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="20ec7053-908f-46d0-8b5e-f3fdb57d433e" containerName="cleanup"
Nov 03 09:15:24.783749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.783150    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="61f3f0ff-1cbf-4e08-8d44-f44952e3b01c" containerName="cleanup"
Nov 03 09:15:24.783749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.783161    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="00db03c4-9076-4e59-a799-a260f34a1e7d" containerName="cleanup"
Nov 03 09:15:24.783749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.783169    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="750847c9-fd9f-4d39-8705-ccea1fd3fc17" containerName="cleanup"
Nov 03 09:15:24.783749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.783179    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="00db03c4-9076-4e59-a799-a260f34a1e7d" containerName="cleanup"
Nov 03 09:15:24.788176 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.788151    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2d37a3-zglmc"
Nov 03 09:15:24.842480 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.842448    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b2d37a3-zglmc"]
Nov 03 09:15:24.885029 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.884612    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0103659f-3024-4869-83f5-4fed041510bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b2d37a3-zglmc\" (UID: \"0103659f-3024-4869-83f5-4fed041510bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2d37a3-zglmc"
Nov 03 09:15:24.885029 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.884697    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0103659f-3024-4869-83f5-4fed041510bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b2d37a3-zglmc\" (UID: \"0103659f-3024-4869-83f5-4fed041510bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2d37a3-zglmc"
Nov 03 09:15:24.885029 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.884778    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nskfn\" (UniqueName: \"kubernetes.io/projected/0103659f-3024-4869-83f5-4fed041510bc-kube-api-access-nskfn\") pod \"maas-api-revoke-sub-8b2d37a3-zglmc\" (UID: \"0103659f-3024-4869-83f5-4fed041510bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2d37a3-zglmc"
Nov 03 09:15:24.888146 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.888121    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:15:24.888304 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.888223    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:15:24.888483 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.888441    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:15:24.891910 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.891873    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:15:24.985903 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.985849    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0103659f-3024-4869-83f5-4fed041510bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b2d37a3-zglmc\" (UID: \"0103659f-3024-4869-83f5-4fed041510bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2d37a3-zglmc"
Nov 03 09:15:24.986127 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.985913    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0103659f-3024-4869-83f5-4fed041510bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b2d37a3-zglmc\" (UID: \"0103659f-3024-4869-83f5-4fed041510bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2d37a3-zglmc"
Nov 03 09:15:24.986127 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.986061    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nskfn\" (UniqueName: \"kubernetes.io/projected/0103659f-3024-4869-83f5-4fed041510bc-kube-api-access-nskfn\") pod \"maas-api-revoke-sub-8b2d37a3-zglmc\" (UID: \"0103659f-3024-4869-83f5-4fed041510bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2d37a3-zglmc"
Nov 03 09:15:24.986480 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.986457    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0103659f-3024-4869-83f5-4fed041510bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b2d37a3-zglmc\" (UID: \"0103659f-3024-4869-83f5-4fed041510bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2d37a3-zglmc"
Nov 03 09:15:24.989556 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:24.989525    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0103659f-3024-4869-83f5-4fed041510bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b2d37a3-zglmc\" (UID: \"0103659f-3024-4869-83f5-4fed041510bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2d37a3-zglmc"
Nov 03 09:15:25.065221 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:25.065139    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nskfn\" (UniqueName: \"kubernetes.io/projected/0103659f-3024-4869-83f5-4fed041510bc-kube-api-access-nskfn\") pod \"maas-api-revoke-sub-8b2d37a3-zglmc\" (UID: \"0103659f-3024-4869-83f5-4fed041510bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2d37a3-zglmc"
Nov 03 09:15:25.105678 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:25.105656    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2d37a3-zglmc"
Nov 03 09:15:25.540633 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:25.525629    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b2d37a3-zglmc"]
Nov 03 09:15:25.540633 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:15:25.539861    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0103659f_3024_4869_83f5_4fed041510bc.slice/crio-55847de88a9bc89a3fcecdd3be5c10f6a98d43626ccf719c903f5b22c0c0ce66 WatchSource:0}: Error finding container 55847de88a9bc89a3fcecdd3be5c10f6a98d43626ccf719c903f5b22c0c0ce66: Status 404 returned error can't find the container with id 55847de88a9bc89a3fcecdd3be5c10f6a98d43626ccf719c903f5b22c0c0ce66
Nov 03 09:15:25.636104 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:25.636060    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2d37a3-zglmc" event={"ID":"0103659f-3024-4869-83f5-4fed041510bc","Type":"ContainerStarted","Data":"55847de88a9bc89a3fcecdd3be5c10f6a98d43626ccf719c903f5b22c0c0ce66"}
Nov 03 09:15:26.643225 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:26.643191    4473 generic.go:356] "Generic (PLEG): container finished" podID="0103659f-3024-4869-83f5-4fed041510bc" containerID="50bcfb98e558e1ba36121c9b6c4c43bf5018b273f174c63daa1435101433fd8f" exitCode=0
Nov 03 09:15:26.643620 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:26.643281    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2d37a3-zglmc" event={"ID":"0103659f-3024-4869-83f5-4fed041510bc","Type":"ContainerDied","Data":"50bcfb98e558e1ba36121c9b6c4c43bf5018b273f174c63daa1435101433fd8f"}
Nov 03 09:15:27.170689 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:27.170660    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"]
Nov 03 09:15:27.649917 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:27.649882    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw" podUID="14dd6c95-a250-41c5-87c7-ae741f96d01a" containerName="istio-proxy" containerID="cri-o://6a3e68602df23f6d3e0fa7b142c264ab2419d360791c94438dbafdf331faf95e" gracePeriod=30
Nov 03 09:15:27.804279 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:27.804259    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2d37a3-zglmc"
Nov 03 09:15:27.920246 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:27.920165    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0103659f-3024-4869-83f5-4fed041510bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0103659f-3024-4869-83f5-4fed041510bc-maas-api-openshift-ca\") pod \"0103659f-3024-4869-83f5-4fed041510bc\" (UID: \"0103659f-3024-4869-83f5-4fed041510bc\") "
Nov 03 09:15:27.920417 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:27.920268    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0103659f-3024-4869-83f5-4fed041510bc-kube-api-access-nskfn\" (UniqueName: \"kubernetes.io/projected/0103659f-3024-4869-83f5-4fed041510bc-kube-api-access-nskfn\") pod \"0103659f-3024-4869-83f5-4fed041510bc\" (UID: \"0103659f-3024-4869-83f5-4fed041510bc\") "
Nov 03 09:15:27.920417 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:27.920379    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0103659f-3024-4869-83f5-4fed041510bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0103659f-3024-4869-83f5-4fed041510bc-maas-api-xks-ca\") pod \"0103659f-3024-4869-83f5-4fed041510bc\" (UID: \"0103659f-3024-4869-83f5-4fed041510bc\") "
Nov 03 09:15:27.920582 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:27.920550    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0103659f-3024-4869-83f5-4fed041510bc-maas-api-openshift-ca" pod "0103659f-3024-4869-83f5-4fed041510bc" (UID: "0103659f-3024-4869-83f5-4fed041510bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:15:27.921252 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:27.920894    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0103659f-3024-4869-83f5-4fed041510bc-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:27.922598 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:27.922578    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0103659f-3024-4869-83f5-4fed041510bc-maas-api-xks-ca" pod "0103659f-3024-4869-83f5-4fed041510bc" (UID: "0103659f-3024-4869-83f5-4fed041510bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:15:27.922745 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:27.922728    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0103659f-3024-4869-83f5-4fed041510bc-kube-api-access-nskfn" pod "0103659f-3024-4869-83f5-4fed041510bc" (UID: "0103659f-3024-4869-83f5-4fed041510bc"). InnerVolumeSpecName "kube-api-access-nskfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:15:28.022410 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:28.022379    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nskfn\" (UniqueName: \"kubernetes.io/projected/0103659f-3024-4869-83f5-4fed041510bc-kube-api-access-nskfn\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:28.022410 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:28.022409    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0103659f-3024-4869-83f5-4fed041510bc-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:28.656475 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:28.656449    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2d37a3-zglmc"
Nov 03 09:15:28.656475 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:28.656473    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b2d37a3-zglmc" event={"ID":"0103659f-3024-4869-83f5-4fed041510bc","Type":"ContainerDied","Data":"55847de88a9bc89a3fcecdd3be5c10f6a98d43626ccf719c903f5b22c0c0ce66"}
Nov 03 09:15:28.656931 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:28.656516    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55847de88a9bc89a3fcecdd3be5c10f6a98d43626ccf719c903f5b22c0c0ce66"
Nov 03 09:15:30.679016 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:30.678975    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-whmmw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 09:15:30.679386 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:30.679031    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw" podUID="14dd6c95-a250-41c5-87c7-ae741f96d01a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 09:15:32.771389 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:32.771347    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"]
Nov 03 09:15:32.771786 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:32.771594    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5" podUID="990b9e6c-86b3-4bed-beee-4a1537a8c379" containerName="istio-proxy" containerID="cri-o://2227037a2f8d66d69790cfe60615c22a2f8ff9f548043bd12848f413697a9968" gracePeriod=30
Nov 03 09:15:32.961979 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:32.959193    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"
Nov 03 09:15:33.088452 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:33.088418    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14dd6c95-a250-41c5-87c7-ae741f96d01a-istio-token\" (UniqueName: \"kubernetes.io/projected/14dd6c95-a250-41c5-87c7-ae741f96d01a-istio-token\") pod \"14dd6c95-a250-41c5-87c7-ae741f96d01a\" (UID: \"14dd6c95-a250-41c5-87c7-ae741f96d01a\") "
Nov 03 09:15:33.088649 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:33.088505    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14dd6c95-a250-41c5-87c7-ae741f96d01a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14dd6c95-a250-41c5-87c7-ae741f96d01a-istiod-ca-cert\") pod \"14dd6c95-a250-41c5-87c7-ae741f96d01a\" (UID: \"14dd6c95-a250-41c5-87c7-ae741f96d01a\") "
Nov 03 09:15:33.088649 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:33.088573    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14dd6c95-a250-41c5-87c7-ae741f96d01a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14dd6c95-a250-41c5-87c7-ae741f96d01a-istio-envoy\") pod \"14dd6c95-a250-41c5-87c7-ae741f96d01a\" (UID: \"14dd6c95-a250-41c5-87c7-ae741f96d01a\") "
Nov 03 09:15:33.088795 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:33.088700    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14dd6c95-a250-41c5-87c7-ae741f96d01a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14dd6c95-a250-41c5-87c7-ae741f96d01a-workload-certs\") pod \"14dd6c95-a250-41c5-87c7-ae741f96d01a\" (UID: \"14dd6c95-a250-41c5-87c7-ae741f96d01a\") "
Nov 03 09:15:33.088795 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:33.088741    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14dd6c95-a250-41c5-87c7-ae741f96d01a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14dd6c95-a250-41c5-87c7-ae741f96d01a-istio-data\") pod \"14dd6c95-a250-41c5-87c7-ae741f96d01a\" (UID: \"14dd6c95-a250-41c5-87c7-ae741f96d01a\") "
Nov 03 09:15:33.088795 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:33.088791    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14dd6c95-a250-41c5-87c7-ae741f96d01a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14dd6c95-a250-41c5-87c7-ae741f96d01a-workload-socket\") pod \"14dd6c95-a250-41c5-87c7-ae741f96d01a\" (UID: \"14dd6c95-a250-41c5-87c7-ae741f96d01a\") "
Nov 03 09:15:33.089003 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:33.088832    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14dd6c95-a250-41c5-87c7-ae741f96d01a-kube-api-access-qzp75\" (UniqueName: \"kubernetes.io/projected/14dd6c95-a250-41c5-87c7-ae741f96d01a-kube-api-access-qzp75\") pod \"14dd6c95-a250-41c5-87c7-ae741f96d01a\" (UID: \"14dd6c95-a250-41c5-87c7-ae741f96d01a\") "
Nov 03 09:15:33.089003 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:33.088899    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14dd6c95-a250-41c5-87c7-ae741f96d01a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14dd6c95-a250-41c5-87c7-ae741f96d01a-istio-podinfo\") pod \"14dd6c95-a250-41c5-87c7-ae741f96d01a\" (UID: \"14dd6c95-a250-41c5-87c7-ae741f96d01a\") "
Nov 03 09:15:33.089003 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:33.088906    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14dd6c95-a250-41c5-87c7-ae741f96d01a-workload-certs" pod "14dd6c95-a250-41c5-87c7-ae741f96d01a" (UID: "14dd6c95-a250-41c5-87c7-ae741f96d01a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:15:33.089003 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:33.088927    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14dd6c95-a250-41c5-87c7-ae741f96d01a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14dd6c95-a250-41c5-87c7-ae741f96d01a-credential-socket\") pod \"14dd6c95-a250-41c5-87c7-ae741f96d01a\" (UID: \"14dd6c95-a250-41c5-87c7-ae741f96d01a\") "
Nov 03 09:15:33.089003 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:33.088929    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14dd6c95-a250-41c5-87c7-ae741f96d01a-istiod-ca-cert" pod "14dd6c95-a250-41c5-87c7-ae741f96d01a" (UID: "14dd6c95-a250-41c5-87c7-ae741f96d01a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:15:33.089283 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:33.089102    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14dd6c95-a250-41c5-87c7-ae741f96d01a-workload-socket" pod "14dd6c95-a250-41c5-87c7-ae741f96d01a" (UID: "14dd6c95-a250-41c5-87c7-ae741f96d01a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:15:33.089426 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:33.089406    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14dd6c95-a250-41c5-87c7-ae741f96d01a-istio-data" pod "14dd6c95-a250-41c5-87c7-ae741f96d01a" (UID: "14dd6c95-a250-41c5-87c7-ae741f96d01a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:15:33.089489 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:33.089437    4473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14dd6c95-a250-41c5-87c7-ae741f96d01a-istiod-ca-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:33.089489 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:33.089459    4473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14dd6c95-a250-41c5-87c7-ae741f96d01a-workload-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:33.089489 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:33.089476    4473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14dd6c95-a250-41c5-87c7-ae741f96d01a-workload-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:33.089703 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:33.089677    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14dd6c95-a250-41c5-87c7-ae741f96d01a-credential-socket" pod "14dd6c95-a250-41c5-87c7-ae741f96d01a" (UID: "14dd6c95-a250-41c5-87c7-ae741f96d01a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:15:33.091072 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:33.091044    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14dd6c95-a250-41c5-87c7-ae741f96d01a-istio-token" pod "14dd6c95-a250-41c5-87c7-ae741f96d01a" (UID: "14dd6c95-a250-41c5-87c7-ae741f96d01a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:15:33.091567 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:33.091545    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14dd6c95-a250-41c5-87c7-ae741f96d01a-kube-api-access-qzp75" pod "14dd6c95-a250-41c5-87c7-ae741f96d01a" (UID: "14dd6c95-a250-41c5-87c7-ae741f96d01a"). InnerVolumeSpecName "kube-api-access-qzp75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:15:33.091662 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:33.091604    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14dd6c95-a250-41c5-87c7-ae741f96d01a-istio-envoy" pod "14dd6c95-a250-41c5-87c7-ae741f96d01a" (UID: "14dd6c95-a250-41c5-87c7-ae741f96d01a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:15:33.091914 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:33.091901    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14dd6c95-a250-41c5-87c7-ae741f96d01a-istio-podinfo" pod "14dd6c95-a250-41c5-87c7-ae741f96d01a" (UID: "14dd6c95-a250-41c5-87c7-ae741f96d01a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 09:15:33.192004 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:33.191904    4473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14dd6c95-a250-41c5-87c7-ae741f96d01a-istio-token\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:33.192004 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:33.191966    4473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14dd6c95-a250-41c5-87c7-ae741f96d01a-istio-envoy\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:33.192004 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:33.191982    4473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14dd6c95-a250-41c5-87c7-ae741f96d01a-istio-data\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:33.192004 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:33.191998    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzp75\" (UniqueName: \"kubernetes.io/projected/14dd6c95-a250-41c5-87c7-ae741f96d01a-kube-api-access-qzp75\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:33.192242 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:33.192014    4473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14dd6c95-a250-41c5-87c7-ae741f96d01a-istio-podinfo\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:33.192242 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:33.192029    4473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14dd6c95-a250-41c5-87c7-ae741f96d01a-credential-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:33.696872 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:33.696838    4473 generic.go:356] "Generic (PLEG): container finished" podID="14dd6c95-a250-41c5-87c7-ae741f96d01a" containerID="6a3e68602df23f6d3e0fa7b142c264ab2419d360791c94438dbafdf331faf95e" exitCode=0
Nov 03 09:15:33.697065 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:33.696889    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"
Nov 03 09:15:33.697065 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:33.696925    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw" event={"ID":"14dd6c95-a250-41c5-87c7-ae741f96d01a","Type":"ContainerDied","Data":"6a3e68602df23f6d3e0fa7b142c264ab2419d360791c94438dbafdf331faf95e"}
Nov 03 09:15:33.697065 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:33.697011    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw" event={"ID":"14dd6c95-a250-41c5-87c7-ae741f96d01a","Type":"ContainerDied","Data":"90288be7bcdc1fedf6db7681904e923254f14d08434e03cd40861fbbb64f3559"}
Nov 03 09:15:33.697065 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:33.697040    4473 scope.go:122] "RemoveContainer" containerID="6a3e68602df23f6d3e0fa7b142c264ab2419d360791c94438dbafdf331faf95e"
Nov 03 09:15:33.708623 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:33.708605    4473 scope.go:122] "RemoveContainer" containerID="6a3e68602df23f6d3e0fa7b142c264ab2419d360791c94438dbafdf331faf95e"
Nov 03 09:15:33.708908 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:15:33.708890    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a3e68602df23f6d3e0fa7b142c264ab2419d360791c94438dbafdf331faf95e\": container with ID starting with 6a3e68602df23f6d3e0fa7b142c264ab2419d360791c94438dbafdf331faf95e not found: ID does not exist" containerID="6a3e68602df23f6d3e0fa7b142c264ab2419d360791c94438dbafdf331faf95e"
Nov 03 09:15:33.708969 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:33.708917    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a3e68602df23f6d3e0fa7b142c264ab2419d360791c94438dbafdf331faf95e"} err="failed to get container status \"6a3e68602df23f6d3e0fa7b142c264ab2419d360791c94438dbafdf331faf95e\": rpc error: code = NotFound desc = could not find container \"6a3e68602df23f6d3e0fa7b142c264ab2419d360791c94438dbafdf331faf95e\": container with ID starting with 6a3e68602df23f6d3e0fa7b142c264ab2419d360791c94438dbafdf331faf95e not found: ID does not exist"
Nov 03 09:15:33.869560 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:33.869533    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"]
Nov 03 09:15:33.919118 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:33.919087    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-whmmw"]
Nov 03 09:15:35.872164 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:35.872133    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14dd6c95-a250-41c5-87c7-ae741f96d01a" path="/var/lib/kubelet/pods/14dd6c95-a250-41c5-87c7-ae741f96d01a/volumes"
Nov 03 09:15:38.035433 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:38.035404    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"
Nov 03 09:15:38.151411 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:38.151378    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/990b9e6c-86b3-4bed-beee-4a1537a8c379-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/990b9e6c-86b3-4bed-beee-4a1537a8c379-istio-podinfo\") pod \"990b9e6c-86b3-4bed-beee-4a1537a8c379\" (UID: \"990b9e6c-86b3-4bed-beee-4a1537a8c379\") "
Nov 03 09:15:38.151566 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:38.151421    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/990b9e6c-86b3-4bed-beee-4a1537a8c379-kube-api-access-htftc\" (UniqueName: \"kubernetes.io/projected/990b9e6c-86b3-4bed-beee-4a1537a8c379-kube-api-access-htftc\") pod \"990b9e6c-86b3-4bed-beee-4a1537a8c379\" (UID: \"990b9e6c-86b3-4bed-beee-4a1537a8c379\") "
Nov 03 09:15:38.151566 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:38.151484    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/990b9e6c-86b3-4bed-beee-4a1537a8c379-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/990b9e6c-86b3-4bed-beee-4a1537a8c379-workload-socket\") pod \"990b9e6c-86b3-4bed-beee-4a1537a8c379\" (UID: \"990b9e6c-86b3-4bed-beee-4a1537a8c379\") "
Nov 03 09:15:38.151566 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:38.151538    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/990b9e6c-86b3-4bed-beee-4a1537a8c379-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/990b9e6c-86b3-4bed-beee-4a1537a8c379-istio-envoy\") pod \"990b9e6c-86b3-4bed-beee-4a1537a8c379\" (UID: \"990b9e6c-86b3-4bed-beee-4a1537a8c379\") "
Nov 03 09:15:38.151886 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:38.151584    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/990b9e6c-86b3-4bed-beee-4a1537a8c379-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/990b9e6c-86b3-4bed-beee-4a1537a8c379-workload-certs\") pod \"990b9e6c-86b3-4bed-beee-4a1537a8c379\" (UID: \"990b9e6c-86b3-4bed-beee-4a1537a8c379\") "
Nov 03 09:15:38.151886 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:38.151637    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/990b9e6c-86b3-4bed-beee-4a1537a8c379-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/990b9e6c-86b3-4bed-beee-4a1537a8c379-istiod-ca-cert\") pod \"990b9e6c-86b3-4bed-beee-4a1537a8c379\" (UID: \"990b9e6c-86b3-4bed-beee-4a1537a8c379\") "
Nov 03 09:15:38.151886 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:38.151718    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/990b9e6c-86b3-4bed-beee-4a1537a8c379-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/990b9e6c-86b3-4bed-beee-4a1537a8c379-credential-socket\") pod \"990b9e6c-86b3-4bed-beee-4a1537a8c379\" (UID: \"990b9e6c-86b3-4bed-beee-4a1537a8c379\") "
Nov 03 09:15:38.151886 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:38.151791    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/990b9e6c-86b3-4bed-beee-4a1537a8c379-istio-token\" (UniqueName: \"kubernetes.io/projected/990b9e6c-86b3-4bed-beee-4a1537a8c379-istio-token\") pod \"990b9e6c-86b3-4bed-beee-4a1537a8c379\" (UID: \"990b9e6c-86b3-4bed-beee-4a1537a8c379\") "
Nov 03 09:15:38.151886 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:38.151829    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/990b9e6c-86b3-4bed-beee-4a1537a8c379-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/990b9e6c-86b3-4bed-beee-4a1537a8c379-istio-data\") pod \"990b9e6c-86b3-4bed-beee-4a1537a8c379\" (UID: \"990b9e6c-86b3-4bed-beee-4a1537a8c379\") "
Nov 03 09:15:38.152187 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:38.151875    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/990b9e6c-86b3-4bed-beee-4a1537a8c379-workload-certs" pod "990b9e6c-86b3-4bed-beee-4a1537a8c379" (UID: "990b9e6c-86b3-4bed-beee-4a1537a8c379"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:15:38.152187 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:38.151888    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/990b9e6c-86b3-4bed-beee-4a1537a8c379-workload-socket" pod "990b9e6c-86b3-4bed-beee-4a1537a8c379" (UID: "990b9e6c-86b3-4bed-beee-4a1537a8c379"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:15:38.152187 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:38.152055    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/990b9e6c-86b3-4bed-beee-4a1537a8c379-credential-socket" pod "990b9e6c-86b3-4bed-beee-4a1537a8c379" (UID: "990b9e6c-86b3-4bed-beee-4a1537a8c379"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:15:38.152187 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:38.152171    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/990b9e6c-86b3-4bed-beee-4a1537a8c379-istiod-ca-cert" pod "990b9e6c-86b3-4bed-beee-4a1537a8c379" (UID: "990b9e6c-86b3-4bed-beee-4a1537a8c379"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:15:38.152385 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:38.152271    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/990b9e6c-86b3-4bed-beee-4a1537a8c379-istio-data" pod "990b9e6c-86b3-4bed-beee-4a1537a8c379" (UID: "990b9e6c-86b3-4bed-beee-4a1537a8c379"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:15:38.152445 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:38.152404    4473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/990b9e6c-86b3-4bed-beee-4a1537a8c379-workload-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:38.152445 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:38.152425    4473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/990b9e6c-86b3-4bed-beee-4a1537a8c379-istiod-ca-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:38.152445 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:38.152435    4473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/990b9e6c-86b3-4bed-beee-4a1537a8c379-credential-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:38.152770 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:38.152447    4473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/990b9e6c-86b3-4bed-beee-4a1537a8c379-istio-data\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:38.152770 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:38.152459    4473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/990b9e6c-86b3-4bed-beee-4a1537a8c379-workload-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:38.154105 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:38.154075    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/990b9e6c-86b3-4bed-beee-4a1537a8c379-istio-envoy" pod "990b9e6c-86b3-4bed-beee-4a1537a8c379" (UID: "990b9e6c-86b3-4bed-beee-4a1537a8c379"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:15:38.154311 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:38.154284    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/990b9e6c-86b3-4bed-beee-4a1537a8c379-istio-token" pod "990b9e6c-86b3-4bed-beee-4a1537a8c379" (UID: "990b9e6c-86b3-4bed-beee-4a1537a8c379"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:15:38.154406 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:38.154362    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/990b9e6c-86b3-4bed-beee-4a1537a8c379-istio-podinfo" pod "990b9e6c-86b3-4bed-beee-4a1537a8c379" (UID: "990b9e6c-86b3-4bed-beee-4a1537a8c379"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 09:15:38.154624 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:38.154457    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/990b9e6c-86b3-4bed-beee-4a1537a8c379-kube-api-access-htftc" pod "990b9e6c-86b3-4bed-beee-4a1537a8c379" (UID: "990b9e6c-86b3-4bed-beee-4a1537a8c379"). InnerVolumeSpecName "kube-api-access-htftc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:15:38.253912 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:38.253877    4473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/990b9e6c-86b3-4bed-beee-4a1537a8c379-istio-envoy\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:38.253912 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:38.253914    4473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/990b9e6c-86b3-4bed-beee-4a1537a8c379-istio-token\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:38.254131 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:38.253929    4473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/990b9e6c-86b3-4bed-beee-4a1537a8c379-istio-podinfo\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:38.254131 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:38.253976    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htftc\" (UniqueName: \"kubernetes.io/projected/990b9e6c-86b3-4bed-beee-4a1537a8c379-kube-api-access-htftc\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:38.732693 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:38.732646    4473 generic.go:356] "Generic (PLEG): container finished" podID="990b9e6c-86b3-4bed-beee-4a1537a8c379" containerID="2227037a2f8d66d69790cfe60615c22a2f8ff9f548043bd12848f413697a9968" exitCode=0
Nov 03 09:15:38.732868 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:38.732783    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"
Nov 03 09:15:38.732868 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:38.732806    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5" event={"ID":"990b9e6c-86b3-4bed-beee-4a1537a8c379","Type":"ContainerDied","Data":"2227037a2f8d66d69790cfe60615c22a2f8ff9f548043bd12848f413697a9968"}
Nov 03 09:15:38.733028 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:38.732868    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5" event={"ID":"990b9e6c-86b3-4bed-beee-4a1537a8c379","Type":"ContainerDied","Data":"706fbe0c476c6737d47087c033ec46d2c2947fcf19e931672be870545e829cd2"}
Nov 03 09:15:38.733028 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:38.732903    4473 scope.go:122] "RemoveContainer" containerID="2227037a2f8d66d69790cfe60615c22a2f8ff9f548043bd12848f413697a9968"
Nov 03 09:15:38.743158 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:38.743136    4473 scope.go:122] "RemoveContainer" containerID="f96d73b89cffe97819b840bd987c4c8756144953435cd01964c79af15f49dc23"
Nov 03 09:15:38.752057 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:38.752041    4473 scope.go:122] "RemoveContainer" containerID="2227037a2f8d66d69790cfe60615c22a2f8ff9f548043bd12848f413697a9968"
Nov 03 09:15:38.752317 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:15:38.752302    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2227037a2f8d66d69790cfe60615c22a2f8ff9f548043bd12848f413697a9968\": container with ID starting with 2227037a2f8d66d69790cfe60615c22a2f8ff9f548043bd12848f413697a9968 not found: ID does not exist" containerID="2227037a2f8d66d69790cfe60615c22a2f8ff9f548043bd12848f413697a9968"
Nov 03 09:15:38.752358 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:38.752326    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2227037a2f8d66d69790cfe60615c22a2f8ff9f548043bd12848f413697a9968"} err="failed to get container status \"2227037a2f8d66d69790cfe60615c22a2f8ff9f548043bd12848f413697a9968\": rpc error: code = NotFound desc = could not find container \"2227037a2f8d66d69790cfe60615c22a2f8ff9f548043bd12848f413697a9968\": container with ID starting with 2227037a2f8d66d69790cfe60615c22a2f8ff9f548043bd12848f413697a9968 not found: ID does not exist"
Nov 03 09:15:38.752358 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:38.752343    4473 scope.go:122] "RemoveContainer" containerID="f96d73b89cffe97819b840bd987c4c8756144953435cd01964c79af15f49dc23"
Nov 03 09:15:38.752602 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:15:38.752584    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f96d73b89cffe97819b840bd987c4c8756144953435cd01964c79af15f49dc23\": container with ID starting with f96d73b89cffe97819b840bd987c4c8756144953435cd01964c79af15f49dc23 not found: ID does not exist" containerID="f96d73b89cffe97819b840bd987c4c8756144953435cd01964c79af15f49dc23"
Nov 03 09:15:38.752650 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:38.752608    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f96d73b89cffe97819b840bd987c4c8756144953435cd01964c79af15f49dc23"} err="failed to get container status \"f96d73b89cffe97819b840bd987c4c8756144953435cd01964c79af15f49dc23\": rpc error: code = NotFound desc = could not find container \"f96d73b89cffe97819b840bd987c4c8756144953435cd01964c79af15f49dc23\": container with ID starting with f96d73b89cffe97819b840bd987c4c8756144953435cd01964c79af15f49dc23 not found: ID does not exist"
Nov 03 09:15:38.956042 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:38.956007    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"]
Nov 03 09:15:39.005038 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:39.004934    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-c78f2b-openshift-default-744d54957c-dmtc5"]
Nov 03 09:15:39.869962 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:39.869922    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="990b9e6c-86b3-4bed-beee-4a1537a8c379" path="/var/lib/kubelet/pods/990b9e6c-86b3-4bed-beee-4a1537a8c379/volumes"
Nov 03 09:15:40.982381 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:40.982350    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d038cef-bm85g"]
Nov 03 09:15:40.982872 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:40.982856    4473 cpu_manager.go:414] "containerMap: removing container" podUID="0103659f-3024-4869-83f5-4fed041510bc" containerName="revoke-keys"
Nov 03 09:15:40.982926 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:40.982874    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0103659f-3024-4869-83f5-4fed041510bc" containerName="revoke-keys"
Nov 03 09:15:40.982926 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:40.982888    4473 cpu_manager.go:414] "containerMap: removing container" podUID="750847c9-fd9f-4d39-8705-ccea1fd3fc17" containerName="cleanup"
Nov 03 09:15:40.982926 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:40.982893    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="750847c9-fd9f-4d39-8705-ccea1fd3fc17" containerName="cleanup"
Nov 03 09:15:40.982926 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:40.982904    4473 cpu_manager.go:414] "containerMap: removing container" podUID="00db03c4-9076-4e59-a799-a260f34a1e7d" containerName="cleanup"
Nov 03 09:15:40.982926 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:40.982909    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00db03c4-9076-4e59-a799-a260f34a1e7d" containerName="cleanup"
Nov 03 09:15:40.982926 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:40.982923    4473 cpu_manager.go:414] "containerMap: removing container" podUID="990b9e6c-86b3-4bed-beee-4a1537a8c379" containerName="istio-proxy"
Nov 03 09:15:40.983132 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:40.982928    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="990b9e6c-86b3-4bed-beee-4a1537a8c379" containerName="istio-proxy"
Nov 03 09:15:40.983132 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:40.982965    4473 cpu_manager.go:414] "containerMap: removing container" podUID="bf19596b-0d54-4f5f-a6d9-433b4c130cbc" containerName="cleanup"
Nov 03 09:15:40.983132 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:40.982971    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf19596b-0d54-4f5f-a6d9-433b4c130cbc" containerName="cleanup"
Nov 03 09:15:40.983132 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:40.982981    4473 cpu_manager.go:414] "containerMap: removing container" podUID="14dd6c95-a250-41c5-87c7-ae741f96d01a" containerName="istio-proxy"
Nov 03 09:15:40.983132 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:40.982985    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14dd6c95-a250-41c5-87c7-ae741f96d01a" containerName="istio-proxy"
Nov 03 09:15:40.983132 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:40.983063    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="0103659f-3024-4869-83f5-4fed041510bc" containerName="revoke-keys"
Nov 03 09:15:40.983132 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:40.983073    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="750847c9-fd9f-4d39-8705-ccea1fd3fc17" containerName="cleanup"
Nov 03 09:15:40.983132 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:40.983083    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf19596b-0d54-4f5f-a6d9-433b4c130cbc" containerName="cleanup"
Nov 03 09:15:40.983132 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:40.983089    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="61f3f0ff-1cbf-4e08-8d44-f44952e3b01c" containerName="cleanup"
Nov 03 09:15:40.983132 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:40.983098    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="5571ff39-4a42-4390-aab3-2f1e60011a4e" containerName="istio-proxy"
Nov 03 09:15:40.983132 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:40.983108    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="990b9e6c-86b3-4bed-beee-4a1537a8c379" containerName="istio-proxy"
Nov 03 09:15:40.983132 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:40.983115    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="990b9e6c-86b3-4bed-beee-4a1537a8c379" containerName="istio-proxy"
Nov 03 09:15:40.983132 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:40.983123    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="14dd6c95-a250-41c5-87c7-ae741f96d01a" containerName="istio-proxy"
Nov 03 09:15:40.987552 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:40.987537    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d038cef-bm85g"
Nov 03 09:15:41.045416 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:41.045385    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d038cef-bm85g"]
Nov 03 09:15:41.091640 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:41.091609    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:15:41.091640 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:41.091615    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:15:41.091640 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:41.091615    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:15:41.091871 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:41.091709    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:15:41.095270 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:41.095250    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42f023db-6a98-4a5f-84d3-c9b30fc68658-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d038cef-bm85g\" (UID: \"42f023db-6a98-4a5f-84d3-c9b30fc68658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d038cef-bm85g"
Nov 03 09:15:41.095338 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:41.095289    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42f023db-6a98-4a5f-84d3-c9b30fc68658-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d038cef-bm85g\" (UID: \"42f023db-6a98-4a5f-84d3-c9b30fc68658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d038cef-bm85g"
Nov 03 09:15:41.095338 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:41.095307    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn7lw\" (UniqueName: \"kubernetes.io/projected/42f023db-6a98-4a5f-84d3-c9b30fc68658-kube-api-access-vn7lw\") pod \"maas-api-revoke-sub-2d038cef-bm85g\" (UID: \"42f023db-6a98-4a5f-84d3-c9b30fc68658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d038cef-bm85g"
Nov 03 09:15:41.196615 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:41.196590    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42f023db-6a98-4a5f-84d3-c9b30fc68658-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d038cef-bm85g\" (UID: \"42f023db-6a98-4a5f-84d3-c9b30fc68658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d038cef-bm85g"
Nov 03 09:15:41.196781 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:41.196634    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42f023db-6a98-4a5f-84d3-c9b30fc68658-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d038cef-bm85g\" (UID: \"42f023db-6a98-4a5f-84d3-c9b30fc68658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d038cef-bm85g"
Nov 03 09:15:41.196781 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:41.196651    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn7lw\" (UniqueName: \"kubernetes.io/projected/42f023db-6a98-4a5f-84d3-c9b30fc68658-kube-api-access-vn7lw\") pod \"maas-api-revoke-sub-2d038cef-bm85g\" (UID: \"42f023db-6a98-4a5f-84d3-c9b30fc68658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d038cef-bm85g"
Nov 03 09:15:41.197229 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:41.197210    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42f023db-6a98-4a5f-84d3-c9b30fc68658-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d038cef-bm85g\" (UID: \"42f023db-6a98-4a5f-84d3-c9b30fc68658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d038cef-bm85g"
Nov 03 09:15:41.199215 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:41.199190    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42f023db-6a98-4a5f-84d3-c9b30fc68658-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d038cef-bm85g\" (UID: \"42f023db-6a98-4a5f-84d3-c9b30fc68658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d038cef-bm85g"
Nov 03 09:15:41.256374 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:41.256323    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn7lw\" (UniqueName: \"kubernetes.io/projected/42f023db-6a98-4a5f-84d3-c9b30fc68658-kube-api-access-vn7lw\") pod \"maas-api-revoke-sub-2d038cef-bm85g\" (UID: \"42f023db-6a98-4a5f-84d3-c9b30fc68658\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d038cef-bm85g"
Nov 03 09:15:41.300367 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:41.300349    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d038cef-bm85g"
Nov 03 09:15:41.680667 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:41.680636    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d038cef-bm85g"]
Nov 03 09:15:41.681361 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:15:41.681334    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42f023db_6a98_4a5f_84d3_c9b30fc68658.slice/crio-e5285b7cc5d6da1874595d2bff238a23969840afb80ca2b446319477a89ef0aa WatchSource:0}: Error finding container e5285b7cc5d6da1874595d2bff238a23969840afb80ca2b446319477a89ef0aa: Status 404 returned error can't find the container with id e5285b7cc5d6da1874595d2bff238a23969840afb80ca2b446319477a89ef0aa
Nov 03 09:15:41.753919 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:41.753890    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d038cef-bm85g" event={"ID":"42f023db-6a98-4a5f-84d3-c9b30fc68658","Type":"ContainerStarted","Data":"e5285b7cc5d6da1874595d2bff238a23969840afb80ca2b446319477a89ef0aa"}
Nov 03 09:15:42.760505 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:42.760471    4473 generic.go:356] "Generic (PLEG): container finished" podID="42f023db-6a98-4a5f-84d3-c9b30fc68658" containerID="0b5dc3cc87494d123400a7955afb6d36b8a8bb18292c2378f292b536180bab45" exitCode=0
Nov 03 09:15:42.761087 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:42.760546    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d038cef-bm85g" event={"ID":"42f023db-6a98-4a5f-84d3-c9b30fc68658","Type":"ContainerDied","Data":"0b5dc3cc87494d123400a7955afb6d36b8a8bb18292c2378f292b536180bab45"}
Nov 03 09:15:43.902866 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:43.902840    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d038cef-bm85g"
Nov 03 09:15:44.024075 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:44.023992    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42f023db-6a98-4a5f-84d3-c9b30fc68658-kube-api-access-vn7lw\" (UniqueName: \"kubernetes.io/projected/42f023db-6a98-4a5f-84d3-c9b30fc68658-kube-api-access-vn7lw\") pod \"42f023db-6a98-4a5f-84d3-c9b30fc68658\" (UID: \"42f023db-6a98-4a5f-84d3-c9b30fc68658\") "
Nov 03 09:15:44.024075 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:44.024069    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42f023db-6a98-4a5f-84d3-c9b30fc68658-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42f023db-6a98-4a5f-84d3-c9b30fc68658-maas-api-openshift-ca\") pod \"42f023db-6a98-4a5f-84d3-c9b30fc68658\" (UID: \"42f023db-6a98-4a5f-84d3-c9b30fc68658\") "
Nov 03 09:15:44.024280 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:44.024175    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42f023db-6a98-4a5f-84d3-c9b30fc68658-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42f023db-6a98-4a5f-84d3-c9b30fc68658-maas-api-xks-ca\") pod \"42f023db-6a98-4a5f-84d3-c9b30fc68658\" (UID: \"42f023db-6a98-4a5f-84d3-c9b30fc68658\") "
Nov 03 09:15:44.024442 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:44.024422    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42f023db-6a98-4a5f-84d3-c9b30fc68658-maas-api-openshift-ca" pod "42f023db-6a98-4a5f-84d3-c9b30fc68658" (UID: "42f023db-6a98-4a5f-84d3-c9b30fc68658"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:15:44.024884 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:44.024789    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42f023db-6a98-4a5f-84d3-c9b30fc68658-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:44.026403 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:44.026379    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42f023db-6a98-4a5f-84d3-c9b30fc68658-maas-api-xks-ca" pod "42f023db-6a98-4a5f-84d3-c9b30fc68658" (UID: "42f023db-6a98-4a5f-84d3-c9b30fc68658"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:15:44.026518 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:44.026502    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42f023db-6a98-4a5f-84d3-c9b30fc68658-kube-api-access-vn7lw" pod "42f023db-6a98-4a5f-84d3-c9b30fc68658" (UID: "42f023db-6a98-4a5f-84d3-c9b30fc68658"). InnerVolumeSpecName "kube-api-access-vn7lw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:15:44.125761 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:44.125733    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn7lw\" (UniqueName: \"kubernetes.io/projected/42f023db-6a98-4a5f-84d3-c9b30fc68658-kube-api-access-vn7lw\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:44.125761 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:44.125759    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42f023db-6a98-4a5f-84d3-c9b30fc68658-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:44.773432 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:44.773402    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d038cef-bm85g"
Nov 03 09:15:44.773432 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:44.773417    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d038cef-bm85g" event={"ID":"42f023db-6a98-4a5f-84d3-c9b30fc68658","Type":"ContainerDied","Data":"e5285b7cc5d6da1874595d2bff238a23969840afb80ca2b446319477a89ef0aa"}
Nov 03 09:15:44.773641 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:44.773465    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5285b7cc5d6da1874595d2bff238a23969840afb80ca2b446319477a89ef0aa"
Nov 03 09:15:45.685675 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:45.685629    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7d51490-2kvg2"]
Nov 03 09:15:45.686362 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:45.686341    4473 cpu_manager.go:414] "containerMap: removing container" podUID="990b9e6c-86b3-4bed-beee-4a1537a8c379" containerName="istio-proxy"
Nov 03 09:15:45.686462 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:45.686364    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="990b9e6c-86b3-4bed-beee-4a1537a8c379" containerName="istio-proxy"
Nov 03 09:15:45.686462 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:45.686383    4473 cpu_manager.go:414] "containerMap: removing container" podUID="42f023db-6a98-4a5f-84d3-c9b30fc68658" containerName="revoke-keys"
Nov 03 09:15:45.686462 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:45.686390    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42f023db-6a98-4a5f-84d3-c9b30fc68658" containerName="revoke-keys"
Nov 03 09:15:45.686621 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:45.686535    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="42f023db-6a98-4a5f-84d3-c9b30fc68658" containerName="revoke-keys"
Nov 03 09:15:45.689509 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:45.689488    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d51490-2kvg2"
Nov 03 09:15:45.751005 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:45.750978    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7d51490-2kvg2"]
Nov 03 09:15:45.769822 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:45.769786    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a5eb55a-8ab5-4d98-b409-87a08427dede-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7d51490-2kvg2\" (UID: \"5a5eb55a-8ab5-4d98-b409-87a08427dede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d51490-2kvg2"
Nov 03 09:15:45.769822 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:45.769833    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a5eb55a-8ab5-4d98-b409-87a08427dede-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7d51490-2kvg2\" (UID: \"5a5eb55a-8ab5-4d98-b409-87a08427dede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d51490-2kvg2"
Nov 03 09:15:45.770053 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:45.770002    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn8p6\" (UniqueName: \"kubernetes.io/projected/5a5eb55a-8ab5-4d98-b409-87a08427dede-kube-api-access-gn8p6\") pod \"maas-api-revoke-sub-b7d51490-2kvg2\" (UID: \"5a5eb55a-8ab5-4d98-b409-87a08427dede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d51490-2kvg2"
Nov 03 09:15:45.797654 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:45.797632    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:15:45.797818 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:45.797756    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:15:45.797818 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:45.797754    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:15:45.797818 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:45.797790    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:15:45.890793 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:45.890745    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn8p6\" (UniqueName: \"kubernetes.io/projected/5a5eb55a-8ab5-4d98-b409-87a08427dede-kube-api-access-gn8p6\") pod \"maas-api-revoke-sub-b7d51490-2kvg2\" (UID: \"5a5eb55a-8ab5-4d98-b409-87a08427dede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d51490-2kvg2"
Nov 03 09:15:45.891073 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:45.891052    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a5eb55a-8ab5-4d98-b409-87a08427dede-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7d51490-2kvg2\" (UID: \"5a5eb55a-8ab5-4d98-b409-87a08427dede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d51490-2kvg2"
Nov 03 09:15:45.891167 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:45.891085    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a5eb55a-8ab5-4d98-b409-87a08427dede-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7d51490-2kvg2\" (UID: \"5a5eb55a-8ab5-4d98-b409-87a08427dede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d51490-2kvg2"
Nov 03 09:15:45.893025 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:45.892972    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a5eb55a-8ab5-4d98-b409-87a08427dede-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7d51490-2kvg2\" (UID: \"5a5eb55a-8ab5-4d98-b409-87a08427dede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d51490-2kvg2"
Nov 03 09:15:45.906186 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:45.906145    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a5eb55a-8ab5-4d98-b409-87a08427dede-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7d51490-2kvg2\" (UID: \"5a5eb55a-8ab5-4d98-b409-87a08427dede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d51490-2kvg2"
Nov 03 09:15:45.951348 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:45.951276    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn8p6\" (UniqueName: \"kubernetes.io/projected/5a5eb55a-8ab5-4d98-b409-87a08427dede-kube-api-access-gn8p6\") pod \"maas-api-revoke-sub-b7d51490-2kvg2\" (UID: \"5a5eb55a-8ab5-4d98-b409-87a08427dede\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d51490-2kvg2"
Nov 03 09:15:46.003742 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:46.003714    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d51490-2kvg2"
Nov 03 09:15:46.387666 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:46.387643    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7d51490-2kvg2"]
Nov 03 09:15:46.388894 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:15:46.388866    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a5eb55a_8ab5_4d98_b409_87a08427dede.slice/crio-f42013828b14055f0702d10adcfbd795f164e315ee92908bf0f759f3366cc74e WatchSource:0}: Error finding container f42013828b14055f0702d10adcfbd795f164e315ee92908bf0f759f3366cc74e: Status 404 returned error can't find the container with id f42013828b14055f0702d10adcfbd795f164e315ee92908bf0f759f3366cc74e
Nov 03 09:15:46.787330 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:46.787302    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d51490-2kvg2" event={"ID":"5a5eb55a-8ab5-4d98-b409-87a08427dede","Type":"ContainerStarted","Data":"80c8612effbeb0fe6f427192f89c3699a110aa349d8fac1ddaf6a66380b7b668"}
Nov 03 09:15:46.787690 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:46.787357    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d51490-2kvg2" event={"ID":"5a5eb55a-8ab5-4d98-b409-87a08427dede","Type":"ContainerStarted","Data":"f42013828b14055f0702d10adcfbd795f164e315ee92908bf0f759f3366cc74e"}
Nov 03 09:15:46.906975 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:46.906894    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d51490-2kvg2" podStartSLOduration=1.9068785940000001 podStartE2EDuration="1.906878594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:15:46.904821362 +0000 UTC m=+2171.173884818" watchObservedRunningTime="2026-11-03 09:15:46.906878594 +0000 UTC m=+2171.175942028"
Nov 03 09:15:46.939190 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:15:46.939152    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:15:47.669600 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.669564    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"]
Nov 03 09:15:47.673562 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.673532    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"
Nov 03 09:15:47.712045 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.712012    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr\" (UID: \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"
Nov 03 09:15:47.712147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.712067    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr\" (UID: \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"
Nov 03 09:15:47.712229 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.712148    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnhpz\" (UniqueName: \"kubernetes.io/projected/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-kube-api-access-pnhpz\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr\" (UID: \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"
Nov 03 09:15:47.712274 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.712252    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr\" (UID: \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"
Nov 03 09:15:47.712325 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.712306    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr\" (UID: \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"
Nov 03 09:15:47.712388 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.712342    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr\" (UID: \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"
Nov 03 09:15:47.712388 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.712371    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr\" (UID: \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"
Nov 03 09:15:47.712625 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.712603    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr\" (UID: \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"
Nov 03 09:15:47.712752 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.712652    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr\" (UID: \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"
Nov 03 09:15:47.719284 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.719263    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"]
Nov 03 09:15:47.722893 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.722879    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"
Nov 03 09:15:47.769654 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.769625    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"]
Nov 03 09:15:47.795777 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.795746    4473 generic.go:356] "Generic (PLEG): container finished" podID="5a5eb55a-8ab5-4d98-b409-87a08427dede" containerID="80c8612effbeb0fe6f427192f89c3699a110aa349d8fac1ddaf6a66380b7b668" exitCode=0
Nov 03 09:15:47.796553 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.796174    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d51490-2kvg2" event={"ID":"5a5eb55a-8ab5-4d98-b409-87a08427dede","Type":"ContainerDied","Data":"80c8612effbeb0fe6f427192f89c3699a110aa349d8fac1ddaf6a66380b7b668"}
Nov 03 09:15:47.800450 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.800432    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"]
Nov 03 09:15:47.813918 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.813889    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d33af3b-997c-4af3-8e58-fa2b1b78135b-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q\" (UID: \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"
Nov 03 09:15:47.814053 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.813936    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr\" (UID: \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"
Nov 03 09:15:47.814121 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.814067    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d33af3b-997c-4af3-8e58-fa2b1b78135b-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q\" (UID: \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"
Nov 03 09:15:47.814177 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.814139    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr\" (UID: \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"
Nov 03 09:15:47.814255 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.814200    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr\" (UID: \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"
Nov 03 09:15:47.814255 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.814218    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnhpz\" (UniqueName: \"kubernetes.io/projected/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-kube-api-access-pnhpz\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr\" (UID: \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"
Nov 03 09:15:47.814584 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.814562    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr\" (UID: \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"
Nov 03 09:15:47.814695 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.814616    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d33af3b-997c-4af3-8e58-fa2b1b78135b-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q\" (UID: \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"
Nov 03 09:15:47.814788 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.814747    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d33af3b-997c-4af3-8e58-fa2b1b78135b-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q\" (UID: \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"
Nov 03 09:15:47.814875 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.814816    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm6bs\" (UniqueName: \"kubernetes.io/projected/9d33af3b-997c-4af3-8e58-fa2b1b78135b-kube-api-access-qm6bs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q\" (UID: \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"
Nov 03 09:15:47.815011 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.814995    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr\" (UID: \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"
Nov 03 09:15:47.815093 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.815048    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d33af3b-997c-4af3-8e58-fa2b1b78135b-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q\" (UID: \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"
Nov 03 09:15:47.815187 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.815136    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d33af3b-997c-4af3-8e58-fa2b1b78135b-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q\" (UID: \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"
Nov 03 09:15:47.815187 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.815177    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr\" (UID: \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"
Nov 03 09:15:47.815403 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.815381    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr\" (UID: \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"
Nov 03 09:15:47.815508 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.815424    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr\" (UID: \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"
Nov 03 09:15:47.815645 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.815516    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr\" (UID: \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"
Nov 03 09:15:47.815645 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.815525    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d33af3b-997c-4af3-8e58-fa2b1b78135b-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q\" (UID: \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"
Nov 03 09:15:47.815799 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.815768    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr\" (UID: \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"
Nov 03 09:15:47.815853 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.815798    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d33af3b-997c-4af3-8e58-fa2b1b78135b-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q\" (UID: \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"
Nov 03 09:15:47.815853 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.815807    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr\" (UID: \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"
Nov 03 09:15:47.816185 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.816165    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr\" (UID: \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"
Nov 03 09:15:47.816281 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.816264    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr\" (UID: \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"
Nov 03 09:15:47.817145 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.817130    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr\" (UID: \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"
Nov 03 09:15:47.817455 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.817437    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr\" (UID: \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"
Nov 03 09:15:47.874464 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.874441    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr\" (UID: \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"
Nov 03 09:15:47.874713 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.874694    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnhpz\" (UniqueName: \"kubernetes.io/projected/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-kube-api-access-pnhpz\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr\" (UID: \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"
Nov 03 09:15:47.916540 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.916509    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d33af3b-997c-4af3-8e58-fa2b1b78135b-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q\" (UID: \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"
Nov 03 09:15:47.916710 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.916550    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d33af3b-997c-4af3-8e58-fa2b1b78135b-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q\" (UID: \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"
Nov 03 09:15:47.916710 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.916572    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d33af3b-997c-4af3-8e58-fa2b1b78135b-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q\" (UID: \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"
Nov 03 09:15:47.916710 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.916593    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d33af3b-997c-4af3-8e58-fa2b1b78135b-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q\" (UID: \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"
Nov 03 09:15:47.916902 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.916747    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d33af3b-997c-4af3-8e58-fa2b1b78135b-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q\" (UID: \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"
Nov 03 09:15:47.916902 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.916780    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d33af3b-997c-4af3-8e58-fa2b1b78135b-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q\" (UID: \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"
Nov 03 09:15:47.917325 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.916814    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm6bs\" (UniqueName: \"kubernetes.io/projected/9d33af3b-997c-4af3-8e58-fa2b1b78135b-kube-api-access-qm6bs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q\" (UID: \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"
Nov 03 09:15:47.917467 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.917372    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d33af3b-997c-4af3-8e58-fa2b1b78135b-workload-certs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q\" (UID: \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"
Nov 03 09:15:47.917467 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.917304    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d33af3b-997c-4af3-8e58-fa2b1b78135b-credential-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q\" (UID: \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"
Nov 03 09:15:47.917602 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.917565    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d33af3b-997c-4af3-8e58-fa2b1b78135b-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q\" (UID: \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"
Nov 03 09:15:47.917701 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.917678    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d33af3b-997c-4af3-8e58-fa2b1b78135b-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q\" (UID: \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"
Nov 03 09:15:47.917896 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.917870    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d33af3b-997c-4af3-8e58-fa2b1b78135b-istio-data\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q\" (UID: \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"
Nov 03 09:15:47.918794 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.918757    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d33af3b-997c-4af3-8e58-fa2b1b78135b-workload-socket\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q\" (UID: \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"
Nov 03 09:15:47.919194 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.919172    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d33af3b-997c-4af3-8e58-fa2b1b78135b-istiod-ca-cert\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q\" (UID: \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"
Nov 03 09:15:47.926046 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.920211    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d33af3b-997c-4af3-8e58-fa2b1b78135b-istio-envoy\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q\" (UID: \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"
Nov 03 09:15:47.926046 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.923531    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d33af3b-997c-4af3-8e58-fa2b1b78135b-istio-podinfo\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q\" (UID: \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"
Nov 03 09:15:47.976027 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.975998    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d33af3b-997c-4af3-8e58-fa2b1b78135b-istio-token\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q\" (UID: \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"
Nov 03 09:15:47.977043 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.977022    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm6bs\" (UniqueName: \"kubernetes.io/projected/9d33af3b-997c-4af3-8e58-fa2b1b78135b-kube-api-access-qm6bs\") pod \"e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q\" (UID: \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\") " pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"
Nov 03 09:15:47.987080 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:47.987062    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"
Nov 03 09:15:48.037300 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:48.037108    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"
Nov 03 09:15:48.419706 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:48.419506    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"]
Nov 03 09:15:48.419854 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:15:48.419751    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bd912d9_a203_42ab_b4e4_22fd099dd1e5.slice/crio-1fd78c7e7879da40a47cb38d0bd5db89a6da76db9b24ad8408fd87bcf1eea595 WatchSource:0}: Error finding container 1fd78c7e7879da40a47cb38d0bd5db89a6da76db9b24ad8408fd87bcf1eea595: Status 404 returned error can't find the container with id 1fd78c7e7879da40a47cb38d0bd5db89a6da76db9b24ad8408fd87bcf1eea595
Nov 03 09:15:48.422295 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:48.422266    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:15:48.422376 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:48.422325    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:15:48.422376 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:48.422352    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:15:48.472902 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:48.472875    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"]
Nov 03 09:15:48.473241 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:15:48.473216    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d33af3b_997c_4af3_8e58_fa2b1b78135b.slice/crio-91f1bbea4180b348fbe5aeb296eb54b0639e686df86c120c9c139124cba7ea81 WatchSource:0}: Error finding container 91f1bbea4180b348fbe5aeb296eb54b0639e686df86c120c9c139124cba7ea81: Status 404 returned error can't find the container with id 91f1bbea4180b348fbe5aeb296eb54b0639e686df86c120c9c139124cba7ea81
Nov 03 09:15:48.475573 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:48.475544    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:15:48.475647 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:48.475603    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:15:48.475647 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:48.475631    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:15:48.804224 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:48.804128    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q" event={"ID":"9d33af3b-997c-4af3-8e58-fa2b1b78135b","Type":"ContainerStarted","Data":"d6e116abaf55b444526eb36dbab67f02257d96b726b66ab66132091f61818abb"}
Nov 03 09:15:48.804224 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:48.804193    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q" event={"ID":"9d33af3b-997c-4af3-8e58-fa2b1b78135b","Type":"ContainerStarted","Data":"91f1bbea4180b348fbe5aeb296eb54b0639e686df86c120c9c139124cba7ea81"}
Nov 03 09:15:48.806297 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:48.806262    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr" event={"ID":"3bd912d9-a203-42ab-b4e4-22fd099dd1e5","Type":"ContainerStarted","Data":"55e1fe8b516abe3ca07578719cf3d75d097688ed575c2bbb0080e308d202a650"}
Nov 03 09:15:48.806418 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:48.806321    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr" event={"ID":"3bd912d9-a203-42ab-b4e4-22fd099dd1e5","Type":"ContainerStarted","Data":"1fd78c7e7879da40a47cb38d0bd5db89a6da76db9b24ad8408fd87bcf1eea595"}
Nov 03 09:15:48.940107 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:48.940052    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q" podStartSLOduration=1.9400366230000001 podStartE2EDuration="1.940036623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:15:48.936287762 +0000 UTC m=+2173.205351225" watchObservedRunningTime="2026-11-03 09:15:48.940036623 +0000 UTC m=+2173.209100056"
Nov 03 09:15:48.987684 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:48.987661    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"
Nov 03 09:15:48.988881 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:48.988864    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d51490-2kvg2"
Nov 03 09:15:49.030202 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:49.030166    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a5eb55a-8ab5-4d98-b409-87a08427dede-kube-api-access-gn8p6\" (UniqueName: \"kubernetes.io/projected/5a5eb55a-8ab5-4d98-b409-87a08427dede-kube-api-access-gn8p6\") pod \"5a5eb55a-8ab5-4d98-b409-87a08427dede\" (UID: \"5a5eb55a-8ab5-4d98-b409-87a08427dede\") "
Nov 03 09:15:49.030382 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:49.030227    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a5eb55a-8ab5-4d98-b409-87a08427dede-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a5eb55a-8ab5-4d98-b409-87a08427dede-maas-api-xks-ca\") pod \"5a5eb55a-8ab5-4d98-b409-87a08427dede\" (UID: \"5a5eb55a-8ab5-4d98-b409-87a08427dede\") "
Nov 03 09:15:49.030382 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:49.030263    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a5eb55a-8ab5-4d98-b409-87a08427dede-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a5eb55a-8ab5-4d98-b409-87a08427dede-maas-api-openshift-ca\") pod \"5a5eb55a-8ab5-4d98-b409-87a08427dede\" (UID: \"5a5eb55a-8ab5-4d98-b409-87a08427dede\") "
Nov 03 09:15:49.030626 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:49.030600    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a5eb55a-8ab5-4d98-b409-87a08427dede-maas-api-openshift-ca" pod "5a5eb55a-8ab5-4d98-b409-87a08427dede" (UID: "5a5eb55a-8ab5-4d98-b409-87a08427dede"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:15:49.030790 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:49.030774    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a5eb55a-8ab5-4d98-b409-87a08427dede-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:49.032416 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:49.032397    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a5eb55a-8ab5-4d98-b409-87a08427dede-kube-api-access-gn8p6" pod "5a5eb55a-8ab5-4d98-b409-87a08427dede" (UID: "5a5eb55a-8ab5-4d98-b409-87a08427dede"). InnerVolumeSpecName "kube-api-access-gn8p6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:15:49.032416 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:49.032402    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a5eb55a-8ab5-4d98-b409-87a08427dede-maas-api-xks-ca" pod "5a5eb55a-8ab5-4d98-b409-87a08427dede" (UID: "5a5eb55a-8ab5-4d98-b409-87a08427dede"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:15:49.037606 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:49.037586    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"
Nov 03 09:15:49.039169 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:49.039149    4473 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 03 09:15:49.039271 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:49.039190    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q" podUID="9d33af3b-997c-4af3-8e58-fa2b1b78135b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 03 09:15:49.057541 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:49.057410    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr" podStartSLOduration=2.057395557 podStartE2EDuration="2.057395557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:15:49.056773861 +0000 UTC m=+2173.325837319" watchObservedRunningTime="2026-11-03 09:15:49.057395557 +0000 UTC m=+2173.326458991"
Nov 03 09:15:49.135971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:49.134867    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn8p6\" (UniqueName: \"kubernetes.io/projected/5a5eb55a-8ab5-4d98-b409-87a08427dede-kube-api-access-gn8p6\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:49.135971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:49.134913    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a5eb55a-8ab5-4d98-b409-87a08427dede-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:15:49.813560 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:49.813536    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d51490-2kvg2"
Nov 03 09:15:49.813959 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:49.813580    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7d51490-2kvg2" event={"ID":"5a5eb55a-8ab5-4d98-b409-87a08427dede","Type":"ContainerDied","Data":"f42013828b14055f0702d10adcfbd795f164e315ee92908bf0f759f3366cc74e"}
Nov 03 09:15:49.813959 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:49.813624    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f42013828b14055f0702d10adcfbd795f164e315ee92908bf0f759f3366cc74e"
Nov 03 09:15:49.989599 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:49.989569    4473 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 09:15:49.989747 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:49.989616    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr" podUID="3bd912d9-a203-42ab-b4e4-22fd099dd1e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 09:15:50.038548 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:50.038512    4473 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 03 09:15:50.038827 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:50.038799    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q" podUID="9d33af3b-997c-4af3-8e58-fa2b1b78135b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 03 09:15:50.987311 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:50.987278    4473 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 09:15:50.987763 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:50.987332    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr" podUID="3bd912d9-a203-42ab-b4e4-22fd099dd1e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 09:15:51.039428 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:51.039391    4473 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 03 09:15:51.039717 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:51.039684    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q" podUID="9d33af3b-997c-4af3-8e58-fa2b1b78135b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 03 09:15:51.988273 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:51.988235    4473 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 09:15:51.988663 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:51.988284    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr" podUID="3bd912d9-a203-42ab-b4e4-22fd099dd1e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 09:15:52.037584 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:52.037545    4473 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 03 09:15:52.038275 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:52.038238    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q" podUID="9d33af3b-997c-4af3-8e58-fa2b1b78135b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 03 09:15:52.990469 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:52.990429    4473 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 09:15:52.991123 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:52.991090    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr" podUID="3bd912d9-a203-42ab-b4e4-22fd099dd1e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 09:15:53.039324 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:53.039286    4473 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 03 09:15:53.040056 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:53.040025    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q" podUID="9d33af3b-997c-4af3-8e58-fa2b1b78135b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 03 09:15:53.263029 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:53.262006    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:15:53.988546 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:53.988512    4473 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 09:15:53.988825 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:53.988799    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr" podUID="3bd912d9-a203-42ab-b4e4-22fd099dd1e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 09:15:54.042359 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:54.042321    4473 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 03 09:15:54.042972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:54.042920    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q" podUID="9d33af3b-997c-4af3-8e58-fa2b1b78135b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 03 09:15:55.001639 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:54.990815    4473 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 09:15:55.001639 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:54.990875    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr" podUID="3bd912d9-a203-42ab-b4e4-22fd099dd1e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 09:15:55.053137 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:55.045331    4473 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 03 09:15:55.053137 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:55.045397    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q" podUID="9d33af3b-997c-4af3-8e58-fa2b1b78135b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 03 09:15:55.990865 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:55.990833    4473 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 09:15:55.991177 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:55.991147    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr" podUID="3bd912d9-a203-42ab-b4e4-22fd099dd1e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 09:15:56.038340 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:56.038303    4473 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 03 09:15:56.038618 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:56.038588    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q" podUID="9d33af3b-997c-4af3-8e58-fa2b1b78135b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 03 09:15:57.059970 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:57.048036    4473 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 09:15:57.059970 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:57.048100    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr" podUID="cc159b5a-6ae2-4f09-a74d-1ff15927d692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:15:57.161444 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:57.161399    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn" podUID="05a8a676-3011-4597-9b71-b70c3387e2b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Nov 03 09:15:57.709140 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:57.709112    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"
Nov 03 09:15:57.709484 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:57.709457    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"
Nov 03 09:15:57.710489 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:57.710469    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"
Nov 03 09:15:57.855531 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:57.855502    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"
Nov 03 09:15:57.855938 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:57.855911    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"
Nov 03 09:15:57.857165 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:15:57.857143    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"
Nov 03 09:16:02.267088 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:02.267057    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f38978da-fmzch"]
Nov 03 09:16:02.267587 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:02.267574    4473 cpu_manager.go:414] "containerMap: removing container" podUID="5a5eb55a-8ab5-4d98-b409-87a08427dede" containerName="revoke-keys"
Nov 03 09:16:02.267634 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:02.267589    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a5eb55a-8ab5-4d98-b409-87a08427dede" containerName="revoke-keys"
Nov 03 09:16:02.267692 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:02.267683    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a5eb55a-8ab5-4d98-b409-87a08427dede" containerName="revoke-keys"
Nov 03 09:16:02.270976 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:02.270961    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f38978da-fmzch"
Nov 03 09:16:02.331876 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:02.331843    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f38978da-fmzch"]
Nov 03 09:16:02.378034 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:02.378002    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:16:02.378034 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:02.378005    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:16:02.378238 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:02.378050    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:16:02.378238 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:02.378140    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:16:02.426443 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:02.426415    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/107cfccf-d2bf-4b2d-a64d-655b12b33fa8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f38978da-fmzch\" (UID: \"107cfccf-d2bf-4b2d-a64d-655b12b33fa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f38978da-fmzch"
Nov 03 09:16:02.426617 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:02.426464    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ghww\" (UniqueName: \"kubernetes.io/projected/107cfccf-d2bf-4b2d-a64d-655b12b33fa8-kube-api-access-5ghww\") pod \"maas-api-revoke-sub-f38978da-fmzch\" (UID: \"107cfccf-d2bf-4b2d-a64d-655b12b33fa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f38978da-fmzch"
Nov 03 09:16:02.426617 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:02.426537    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/107cfccf-d2bf-4b2d-a64d-655b12b33fa8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f38978da-fmzch\" (UID: \"107cfccf-d2bf-4b2d-a64d-655b12b33fa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f38978da-fmzch"
Nov 03 09:16:02.527934 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:02.527845    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/107cfccf-d2bf-4b2d-a64d-655b12b33fa8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f38978da-fmzch\" (UID: \"107cfccf-d2bf-4b2d-a64d-655b12b33fa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f38978da-fmzch"
Nov 03 09:16:02.527934 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:02.527899    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ghww\" (UniqueName: \"kubernetes.io/projected/107cfccf-d2bf-4b2d-a64d-655b12b33fa8-kube-api-access-5ghww\") pod \"maas-api-revoke-sub-f38978da-fmzch\" (UID: \"107cfccf-d2bf-4b2d-a64d-655b12b33fa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f38978da-fmzch"
Nov 03 09:16:02.527934 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:02.527921    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/107cfccf-d2bf-4b2d-a64d-655b12b33fa8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f38978da-fmzch\" (UID: \"107cfccf-d2bf-4b2d-a64d-655b12b33fa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f38978da-fmzch"
Nov 03 09:16:02.528622 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:02.528602    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/107cfccf-d2bf-4b2d-a64d-655b12b33fa8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f38978da-fmzch\" (UID: \"107cfccf-d2bf-4b2d-a64d-655b12b33fa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f38978da-fmzch"
Nov 03 09:16:02.530436 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:02.530412    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/107cfccf-d2bf-4b2d-a64d-655b12b33fa8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f38978da-fmzch\" (UID: \"107cfccf-d2bf-4b2d-a64d-655b12b33fa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f38978da-fmzch"
Nov 03 09:16:02.587101 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:02.587063    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ghww\" (UniqueName: \"kubernetes.io/projected/107cfccf-d2bf-4b2d-a64d-655b12b33fa8-kube-api-access-5ghww\") pod \"maas-api-revoke-sub-f38978da-fmzch\" (UID: \"107cfccf-d2bf-4b2d-a64d-655b12b33fa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f38978da-fmzch"
Nov 03 09:16:02.884491 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:02.884459    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f38978da-fmzch"
Nov 03 09:16:03.262221 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:03.262190    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f38978da-fmzch"]
Nov 03 09:16:03.262649 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:16:03.262624    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod107cfccf_d2bf_4b2d_a64d_655b12b33fa8.slice/crio-f89da17783164c62764d4fbf5c3f22fed0ae52ce036538295644e3439ef15ce5 WatchSource:0}: Error finding container f89da17783164c62764d4fbf5c3f22fed0ae52ce036538295644e3439ef15ce5: Status 404 returned error can't find the container with id f89da17783164c62764d4fbf5c3f22fed0ae52ce036538295644e3439ef15ce5
Nov 03 09:16:04.087795 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:04.087761    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f38978da-fmzch" event={"ID":"107cfccf-d2bf-4b2d-a64d-655b12b33fa8","Type":"ContainerStarted","Data":"42706b8f40348817bf272da07590a25f11fd73afa8e53ca50560f9cfc0454a99"}
Nov 03 09:16:04.088179 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:04.087822    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f38978da-fmzch" event={"ID":"107cfccf-d2bf-4b2d-a64d-655b12b33fa8","Type":"ContainerStarted","Data":"f89da17783164c62764d4fbf5c3f22fed0ae52ce036538295644e3439ef15ce5"}
Nov 03 09:16:04.206787 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:04.206727    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f38978da-fmzch" podStartSLOduration=2.206712333 podStartE2EDuration="2.206712333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:16:04.20372546 +0000 UTC m=+2188.472788924" watchObservedRunningTime="2026-11-03 09:16:04.206712333 +0000 UTC m=+2188.475775768"
Nov 03 09:16:04.440469 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:04.440436    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-8694bd86f9-4vb49"]
Nov 03 09:16:04.444416 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:04.444398    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-8694bd86f9-4vb49"
Nov 03 09:16:04.463808 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:04.463780    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvmkl\" (UniqueName: \"kubernetes.io/projected/92cc6b00-3639-4fdf-beab-510019250b68-kube-api-access-wvmkl\") pod \"payload-pre-processing-e2e-ipp-b-121ee6-8694bd86f9-4vb49\" (UID: \"92cc6b00-3639-4fdf-beab-510019250b68\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-8694bd86f9-4vb49"
Nov 03 09:16:04.463998 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:04.463984    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92cc6b00-3639-4fdf-beab-510019250b68-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-121ee6-8694bd86f9-4vb49\" (UID: \"92cc6b00-3639-4fdf-beab-510019250b68\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-8694bd86f9-4vb49"
Nov 03 09:16:04.500445 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:04.500397    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-121ee6-dockercfg-gqjg6\""
Nov 03 09:16:04.500620 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:04.500603    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-121ee6\""
Nov 03 09:16:04.503050 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:04.503028    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-8694bd86f9-4vb49"]
Nov 03 09:16:04.564574 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:04.564542    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvmkl\" (UniqueName: \"kubernetes.io/projected/92cc6b00-3639-4fdf-beab-510019250b68-kube-api-access-wvmkl\") pod \"payload-pre-processing-e2e-ipp-b-121ee6-8694bd86f9-4vb49\" (UID: \"92cc6b00-3639-4fdf-beab-510019250b68\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-8694bd86f9-4vb49"
Nov 03 09:16:04.564739 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:04.564715    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92cc6b00-3639-4fdf-beab-510019250b68-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-121ee6-8694bd86f9-4vb49\" (UID: \"92cc6b00-3639-4fdf-beab-510019250b68\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-8694bd86f9-4vb49"
Nov 03 09:16:04.565441 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:04.565420    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92cc6b00-3639-4fdf-beab-510019250b68-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-121ee6-8694bd86f9-4vb49\" (UID: \"92cc6b00-3639-4fdf-beab-510019250b68\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-8694bd86f9-4vb49"
Nov 03 09:16:04.624052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:04.624025    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvmkl\" (UniqueName: \"kubernetes.io/projected/92cc6b00-3639-4fdf-beab-510019250b68-kube-api-access-wvmkl\") pod \"payload-pre-processing-e2e-ipp-b-121ee6-8694bd86f9-4vb49\" (UID: \"92cc6b00-3639-4fdf-beab-510019250b68\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-8694bd86f9-4vb49"
Nov 03 09:16:04.757293 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:04.757218    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-8694bd86f9-4vb49"
Nov 03 09:16:05.017047 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:05.016978    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-121ee6-6b46c6847b-qsf5t"]
Nov 03 09:16:05.021920 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:05.021900    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-121ee6-6b46c6847b-qsf5t"
Nov 03 09:16:05.070110 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:05.070075    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c47f7a0-c0b1-4076-b122-ff49eb32fb60-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-121ee6-6b46c6847b-qsf5t\" (UID: \"9c47f7a0-c0b1-4076-b122-ff49eb32fb60\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-121ee6-6b46c6847b-qsf5t"
Nov 03 09:16:05.070262 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:05.070230    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bplfn\" (UniqueName: \"kubernetes.io/projected/9c47f7a0-c0b1-4076-b122-ff49eb32fb60-kube-api-access-bplfn\") pod \"payload-processing-e2e-ipp-b-121ee6-6b46c6847b-qsf5t\" (UID: \"9c47f7a0-c0b1-4076-b122-ff49eb32fb60\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-121ee6-6b46c6847b-qsf5t"
Nov 03 09:16:05.079112 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:05.079073    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-121ee6-6b46c6847b-qsf5t"]
Nov 03 09:16:05.096859 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:05.096827    4473 generic.go:356] "Generic (PLEG): container finished" podID="107cfccf-d2bf-4b2d-a64d-655b12b33fa8" containerID="42706b8f40348817bf272da07590a25f11fd73afa8e53ca50560f9cfc0454a99" exitCode=0
Nov 03 09:16:05.097273 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:05.096919    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f38978da-fmzch" event={"ID":"107cfccf-d2bf-4b2d-a64d-655b12b33fa8","Type":"ContainerDied","Data":"42706b8f40348817bf272da07590a25f11fd73afa8e53ca50560f9cfc0454a99"}
Nov 03 09:16:05.137506 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:05.137480    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-8694bd86f9-4vb49"]
Nov 03 09:16:05.138094 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:16:05.138071    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92cc6b00_3639_4fdf_beab_510019250b68.slice/crio-80998db9d14aec720f2287944cef476f6447d20c6d67cd0a3323af75896b1f73 WatchSource:0}: Error finding container 80998db9d14aec720f2287944cef476f6447d20c6d67cd0a3323af75896b1f73: Status 404 returned error can't find the container with id 80998db9d14aec720f2287944cef476f6447d20c6d67cd0a3323af75896b1f73
Nov 03 09:16:05.171586 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:05.171551    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c47f7a0-c0b1-4076-b122-ff49eb32fb60-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-121ee6-6b46c6847b-qsf5t\" (UID: \"9c47f7a0-c0b1-4076-b122-ff49eb32fb60\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-121ee6-6b46c6847b-qsf5t"
Nov 03 09:16:05.171772 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:05.171755    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bplfn\" (UniqueName: \"kubernetes.io/projected/9c47f7a0-c0b1-4076-b122-ff49eb32fb60-kube-api-access-bplfn\") pod \"payload-processing-e2e-ipp-b-121ee6-6b46c6847b-qsf5t\" (UID: \"9c47f7a0-c0b1-4076-b122-ff49eb32fb60\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-121ee6-6b46c6847b-qsf5t"
Nov 03 09:16:05.172285 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:05.172267    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c47f7a0-c0b1-4076-b122-ff49eb32fb60-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-121ee6-6b46c6847b-qsf5t\" (UID: \"9c47f7a0-c0b1-4076-b122-ff49eb32fb60\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-121ee6-6b46c6847b-qsf5t"
Nov 03 09:16:05.231656 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:05.231613    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bplfn\" (UniqueName: \"kubernetes.io/projected/9c47f7a0-c0b1-4076-b122-ff49eb32fb60-kube-api-access-bplfn\") pod \"payload-processing-e2e-ipp-b-121ee6-6b46c6847b-qsf5t\" (UID: \"9c47f7a0-c0b1-4076-b122-ff49eb32fb60\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-121ee6-6b46c6847b-qsf5t"
Nov 03 09:16:05.335517 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:05.335431    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-121ee6-6b46c6847b-qsf5t"
Nov 03 09:16:05.719096 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:05.719070    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-121ee6-6b46c6847b-qsf5t"]
Nov 03 09:16:05.719800 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:16:05.719769    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c47f7a0_c0b1_4076_b122_ff49eb32fb60.slice/crio-6d7473c415c11d0404f81dce133ece7fc8479291d2206dc16bbdba2c99044845 WatchSource:0}: Error finding container 6d7473c415c11d0404f81dce133ece7fc8479291d2206dc16bbdba2c99044845: Status 404 returned error can't find the container with id 6d7473c415c11d0404f81dce133ece7fc8479291d2206dc16bbdba2c99044845
Nov 03 09:16:06.105592 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:06.105496    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-8694bd86f9-4vb49" event={"ID":"92cc6b00-3639-4fdf-beab-510019250b68","Type":"ContainerStarted","Data":"197b48e27a56f22f853e8895a41616856c4ac7a93e189e09eb89aa9017b69408"}
Nov 03 09:16:06.105592 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:06.105559    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-8694bd86f9-4vb49" event={"ID":"92cc6b00-3639-4fdf-beab-510019250b68","Type":"ContainerStarted","Data":"80998db9d14aec720f2287944cef476f6447d20c6d67cd0a3323af75896b1f73"}
Nov 03 09:16:06.106138 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:06.105609    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-8694bd86f9-4vb49"
Nov 03 09:16:06.107249 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:06.107226    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-121ee6-6b46c6847b-qsf5t" event={"ID":"9c47f7a0-c0b1-4076-b122-ff49eb32fb60","Type":"ContainerStarted","Data":"d32dc9ff519012d63a8c42753a1036789bd4266cc415b40e2b2fcae9bf6abdf2"}
Nov 03 09:16:06.107365 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:06.107268    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-121ee6-6b46c6847b-qsf5t" event={"ID":"9c47f7a0-c0b1-4076-b122-ff49eb32fb60","Type":"ContainerStarted","Data":"6d7473c415c11d0404f81dce133ece7fc8479291d2206dc16bbdba2c99044845"}
Nov 03 09:16:06.107365 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:06.107296    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-121ee6-6b46c6847b-qsf5t"
Nov 03 09:16:06.278261 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:06.278174    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-8694bd86f9-4vb49" podStartSLOduration=2.27815648 podStartE2EDuration="2.27815648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:16:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:16:06.277234832 +0000 UTC m=+2190.546298295" watchObservedRunningTime="2026-11-03 09:16:06.27815648 +0000 UTC m=+2190.547219916"
Nov 03 09:16:06.326749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:06.326724    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f38978da-fmzch"
Nov 03 09:16:06.397374 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:06.397314    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-121ee6-6b46c6847b-qsf5t" podStartSLOduration=2.397292983 podStartE2EDuration="2.397292983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:16:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:16:06.394217998 +0000 UTC m=+2190.663281452" watchObservedRunningTime="2026-11-03 09:16:06.397292983 +0000 UTC m=+2190.666356421"
Nov 03 09:16:06.481700 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:06.481672    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/107cfccf-d2bf-4b2d-a64d-655b12b33fa8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/107cfccf-d2bf-4b2d-a64d-655b12b33fa8-maas-api-openshift-ca\") pod \"107cfccf-d2bf-4b2d-a64d-655b12b33fa8\" (UID: \"107cfccf-d2bf-4b2d-a64d-655b12b33fa8\") "
Nov 03 09:16:06.481863 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:06.481711    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/107cfccf-d2bf-4b2d-a64d-655b12b33fa8-kube-api-access-5ghww\" (UniqueName: \"kubernetes.io/projected/107cfccf-d2bf-4b2d-a64d-655b12b33fa8-kube-api-access-5ghww\") pod \"107cfccf-d2bf-4b2d-a64d-655b12b33fa8\" (UID: \"107cfccf-d2bf-4b2d-a64d-655b12b33fa8\") "
Nov 03 09:16:06.481863 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:06.481747    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/107cfccf-d2bf-4b2d-a64d-655b12b33fa8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/107cfccf-d2bf-4b2d-a64d-655b12b33fa8-maas-api-xks-ca\") pod \"107cfccf-d2bf-4b2d-a64d-655b12b33fa8\" (UID: \"107cfccf-d2bf-4b2d-a64d-655b12b33fa8\") "
Nov 03 09:16:06.482176 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:06.482147    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/107cfccf-d2bf-4b2d-a64d-655b12b33fa8-maas-api-openshift-ca" pod "107cfccf-d2bf-4b2d-a64d-655b12b33fa8" (UID: "107cfccf-d2bf-4b2d-a64d-655b12b33fa8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:16:06.482361 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:06.482342    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/107cfccf-d2bf-4b2d-a64d-655b12b33fa8-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:16:06.484076 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:06.484052    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/107cfccf-d2bf-4b2d-a64d-655b12b33fa8-kube-api-access-5ghww" pod "107cfccf-d2bf-4b2d-a64d-655b12b33fa8" (UID: "107cfccf-d2bf-4b2d-a64d-655b12b33fa8"). InnerVolumeSpecName "kube-api-access-5ghww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:16:06.484155 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:06.484108    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/107cfccf-d2bf-4b2d-a64d-655b12b33fa8-maas-api-xks-ca" pod "107cfccf-d2bf-4b2d-a64d-655b12b33fa8" (UID: "107cfccf-d2bf-4b2d-a64d-655b12b33fa8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:16:06.583979 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:06.583908    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ghww\" (UniqueName: \"kubernetes.io/projected/107cfccf-d2bf-4b2d-a64d-655b12b33fa8-kube-api-access-5ghww\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:16:06.583979 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:06.583959    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/107cfccf-d2bf-4b2d-a64d-655b12b33fa8-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:16:07.115026 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:07.114991    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f38978da-fmzch" event={"ID":"107cfccf-d2bf-4b2d-a64d-655b12b33fa8","Type":"ContainerDied","Data":"f89da17783164c62764d4fbf5c3f22fed0ae52ce036538295644e3439ef15ce5"}
Nov 03 09:16:07.115026 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:07.115044    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f89da17783164c62764d4fbf5c3f22fed0ae52ce036538295644e3439ef15ce5"
Nov 03 09:16:07.115490 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:07.115000    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f38978da-fmzch"
Nov 03 09:16:08.152847 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:08.152807    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0591542-ggbkb"]
Nov 03 09:16:08.153375 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:08.153304    4473 cpu_manager.go:414] "containerMap: removing container" podUID="107cfccf-d2bf-4b2d-a64d-655b12b33fa8" containerName="revoke-keys"
Nov 03 09:16:08.153375 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:08.153316    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="107cfccf-d2bf-4b2d-a64d-655b12b33fa8" containerName="revoke-keys"
Nov 03 09:16:08.153506 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:08.153445    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="107cfccf-d2bf-4b2d-a64d-655b12b33fa8" containerName="revoke-keys"
Nov 03 09:16:08.156806 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:08.156785    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0591542-ggbkb"
Nov 03 09:16:08.198869 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:08.198830    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f8b6605-ed6f-494d-968f-bbd776b7057d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0591542-ggbkb\" (UID: \"4f8b6605-ed6f-494d-968f-bbd776b7057d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0591542-ggbkb"
Nov 03 09:16:08.199045 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:08.198877    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f8b6605-ed6f-494d-968f-bbd776b7057d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0591542-ggbkb\" (UID: \"4f8b6605-ed6f-494d-968f-bbd776b7057d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0591542-ggbkb"
Nov 03 09:16:08.199119 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:08.199094    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsxn2\" (UniqueName: \"kubernetes.io/projected/4f8b6605-ed6f-494d-968f-bbd776b7057d-kube-api-access-wsxn2\") pod \"maas-api-revoke-sub-c0591542-ggbkb\" (UID: \"4f8b6605-ed6f-494d-968f-bbd776b7057d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0591542-ggbkb"
Nov 03 09:16:08.214842 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:08.214810    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0591542-ggbkb"]
Nov 03 09:16:08.261142 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:08.261102    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:16:08.261142 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:08.261114    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:16:08.261323 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:08.261161    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:16:08.261323 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:08.261169    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:16:08.299832 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:08.299807    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f8b6605-ed6f-494d-968f-bbd776b7057d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0591542-ggbkb\" (UID: \"4f8b6605-ed6f-494d-968f-bbd776b7057d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0591542-ggbkb"
Nov 03 09:16:08.300004 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:08.299843    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f8b6605-ed6f-494d-968f-bbd776b7057d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0591542-ggbkb\" (UID: \"4f8b6605-ed6f-494d-968f-bbd776b7057d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0591542-ggbkb"
Nov 03 09:16:08.300004 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:08.299914    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsxn2\" (UniqueName: \"kubernetes.io/projected/4f8b6605-ed6f-494d-968f-bbd776b7057d-kube-api-access-wsxn2\") pod \"maas-api-revoke-sub-c0591542-ggbkb\" (UID: \"4f8b6605-ed6f-494d-968f-bbd776b7057d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0591542-ggbkb"
Nov 03 09:16:08.300555 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:08.300535    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f8b6605-ed6f-494d-968f-bbd776b7057d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0591542-ggbkb\" (UID: \"4f8b6605-ed6f-494d-968f-bbd776b7057d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0591542-ggbkb"
Nov 03 09:16:08.302394 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:08.302371    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f8b6605-ed6f-494d-968f-bbd776b7057d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0591542-ggbkb\" (UID: \"4f8b6605-ed6f-494d-968f-bbd776b7057d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0591542-ggbkb"
Nov 03 09:16:08.359465 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:08.359436    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsxn2\" (UniqueName: \"kubernetes.io/projected/4f8b6605-ed6f-494d-968f-bbd776b7057d-kube-api-access-wsxn2\") pod \"maas-api-revoke-sub-c0591542-ggbkb\" (UID: \"4f8b6605-ed6f-494d-968f-bbd776b7057d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0591542-ggbkb"
Nov 03 09:16:08.477413 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:08.477343    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0591542-ggbkb"
Nov 03 09:16:08.861843 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:08.861821    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0591542-ggbkb"]
Nov 03 09:16:08.862687 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:16:08.862661    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f8b6605_ed6f_494d_968f_bbd776b7057d.slice/crio-173dbacb73334bbcfb249fb2fec01a44305503472750963ef55cd37a1f816364 WatchSource:0}: Error finding container 173dbacb73334bbcfb249fb2fec01a44305503472750963ef55cd37a1f816364: Status 404 returned error can't find the container with id 173dbacb73334bbcfb249fb2fec01a44305503472750963ef55cd37a1f816364
Nov 03 09:16:09.128136 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:09.128108    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0591542-ggbkb" event={"ID":"4f8b6605-ed6f-494d-968f-bbd776b7057d","Type":"ContainerStarted","Data":"173dbacb73334bbcfb249fb2fec01a44305503472750963ef55cd37a1f816364"}
Nov 03 09:16:10.135354 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:10.135321    4473 generic.go:356] "Generic (PLEG): container finished" podID="4f8b6605-ed6f-494d-968f-bbd776b7057d" containerID="c98ebdb265367e64dea64a716e1b5814f219a901a93d71e3a05a7e21b4f2df13" exitCode=0
Nov 03 09:16:10.135761 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:10.135407    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0591542-ggbkb" event={"ID":"4f8b6605-ed6f-494d-968f-bbd776b7057d","Type":"ContainerDied","Data":"c98ebdb265367e64dea64a716e1b5814f219a901a93d71e3a05a7e21b4f2df13"}
Nov 03 09:16:11.329423 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:11.329387    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0591542-ggbkb"
Nov 03 09:16:11.436029 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:11.435922    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f8b6605-ed6f-494d-968f-bbd776b7057d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f8b6605-ed6f-494d-968f-bbd776b7057d-maas-api-openshift-ca\") pod \"4f8b6605-ed6f-494d-968f-bbd776b7057d\" (UID: \"4f8b6605-ed6f-494d-968f-bbd776b7057d\") "
Nov 03 09:16:11.436191 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:11.436065    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f8b6605-ed6f-494d-968f-bbd776b7057d-kube-api-access-wsxn2\" (UniqueName: \"kubernetes.io/projected/4f8b6605-ed6f-494d-968f-bbd776b7057d-kube-api-access-wsxn2\") pod \"4f8b6605-ed6f-494d-968f-bbd776b7057d\" (UID: \"4f8b6605-ed6f-494d-968f-bbd776b7057d\") "
Nov 03 09:16:11.436191 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:11.436095    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f8b6605-ed6f-494d-968f-bbd776b7057d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f8b6605-ed6f-494d-968f-bbd776b7057d-maas-api-xks-ca\") pod \"4f8b6605-ed6f-494d-968f-bbd776b7057d\" (UID: \"4f8b6605-ed6f-494d-968f-bbd776b7057d\") "
Nov 03 09:16:11.436337 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:11.436307    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f8b6605-ed6f-494d-968f-bbd776b7057d-maas-api-openshift-ca" pod "4f8b6605-ed6f-494d-968f-bbd776b7057d" (UID: "4f8b6605-ed6f-494d-968f-bbd776b7057d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:16:11.436446 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:11.436430    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f8b6605-ed6f-494d-968f-bbd776b7057d-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:16:11.438274 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:11.438251    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f8b6605-ed6f-494d-968f-bbd776b7057d-kube-api-access-wsxn2" pod "4f8b6605-ed6f-494d-968f-bbd776b7057d" (UID: "4f8b6605-ed6f-494d-968f-bbd776b7057d"). InnerVolumeSpecName "kube-api-access-wsxn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:16:11.438274 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:11.438266    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f8b6605-ed6f-494d-968f-bbd776b7057d-maas-api-xks-ca" pod "4f8b6605-ed6f-494d-968f-bbd776b7057d" (UID: "4f8b6605-ed6f-494d-968f-bbd776b7057d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:16:11.538076 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:11.538044    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsxn2\" (UniqueName: \"kubernetes.io/projected/4f8b6605-ed6f-494d-968f-bbd776b7057d-kube-api-access-wsxn2\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:16:11.538076 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:11.538071    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f8b6605-ed6f-494d-968f-bbd776b7057d-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:16:12.149289 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:12.149204    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0591542-ggbkb" event={"ID":"4f8b6605-ed6f-494d-968f-bbd776b7057d","Type":"ContainerDied","Data":"173dbacb73334bbcfb249fb2fec01a44305503472750963ef55cd37a1f816364"}
Nov 03 09:16:12.149289 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:12.149236    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0591542-ggbkb"
Nov 03 09:16:12.149289 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:12.149259    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="173dbacb73334bbcfb249fb2fec01a44305503472750963ef55cd37a1f816364"
Nov 03 09:16:16.943838 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:16:16.943755    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:16:17.117082 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:17.117054    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-121ee6-6b46c6847b-qsf5t"
Nov 03 09:16:17.117243 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:17.117132    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-8694bd86f9-4vb49"
Nov 03 09:16:19.966107 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:19.966076    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01640f00-7pfvq"]
Nov 03 09:16:19.966794 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:19.966773    4473 cpu_manager.go:414] "containerMap: removing container" podUID="4f8b6605-ed6f-494d-968f-bbd776b7057d" containerName="revoke-keys"
Nov 03 09:16:19.966898 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:19.966797    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f8b6605-ed6f-494d-968f-bbd776b7057d" containerName="revoke-keys"
Nov 03 09:16:19.967004 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:19.966990    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f8b6605-ed6f-494d-968f-bbd776b7057d" containerName="revoke-keys"
Nov 03 09:16:19.970511 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:19.970491    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01640f00-7pfvq"
Nov 03 09:16:20.028250 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:20.028221    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01640f00-7pfvq"]
Nov 03 09:16:20.038037 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:20.038006    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/212f6063-6386-4d83-9da1-3030eb8c1482-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01640f00-7pfvq\" (UID: \"212f6063-6386-4d83-9da1-3030eb8c1482\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01640f00-7pfvq"
Nov 03 09:16:20.038164 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:20.038133    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvdhl\" (UniqueName: \"kubernetes.io/projected/212f6063-6386-4d83-9da1-3030eb8c1482-kube-api-access-bvdhl\") pod \"maas-api-revoke-sub-01640f00-7pfvq\" (UID: \"212f6063-6386-4d83-9da1-3030eb8c1482\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01640f00-7pfvq"
Nov 03 09:16:20.038226 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:20.038209    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/212f6063-6386-4d83-9da1-3030eb8c1482-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01640f00-7pfvq\" (UID: \"212f6063-6386-4d83-9da1-3030eb8c1482\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01640f00-7pfvq"
Nov 03 09:16:20.078274 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:20.078242    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:16:20.078700 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:20.078678    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:16:20.079134 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:20.079114    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:16:20.088961 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:20.088780    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:16:20.172984 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:20.171450    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvdhl\" (UniqueName: \"kubernetes.io/projected/212f6063-6386-4d83-9da1-3030eb8c1482-kube-api-access-bvdhl\") pod \"maas-api-revoke-sub-01640f00-7pfvq\" (UID: \"212f6063-6386-4d83-9da1-3030eb8c1482\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01640f00-7pfvq"
Nov 03 09:16:20.172984 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:20.171531    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/212f6063-6386-4d83-9da1-3030eb8c1482-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01640f00-7pfvq\" (UID: \"212f6063-6386-4d83-9da1-3030eb8c1482\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01640f00-7pfvq"
Nov 03 09:16:20.172984 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:20.171729    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/212f6063-6386-4d83-9da1-3030eb8c1482-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01640f00-7pfvq\" (UID: \"212f6063-6386-4d83-9da1-3030eb8c1482\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01640f00-7pfvq"
Nov 03 09:16:20.172984 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:20.172806    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/212f6063-6386-4d83-9da1-3030eb8c1482-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01640f00-7pfvq\" (UID: \"212f6063-6386-4d83-9da1-3030eb8c1482\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01640f00-7pfvq"
Nov 03 09:16:20.181451 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:20.181408    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/212f6063-6386-4d83-9da1-3030eb8c1482-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01640f00-7pfvq\" (UID: \"212f6063-6386-4d83-9da1-3030eb8c1482\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01640f00-7pfvq"
Nov 03 09:16:20.234148 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:20.234062    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvdhl\" (UniqueName: \"kubernetes.io/projected/212f6063-6386-4d83-9da1-3030eb8c1482-kube-api-access-bvdhl\") pod \"maas-api-revoke-sub-01640f00-7pfvq\" (UID: \"212f6063-6386-4d83-9da1-3030eb8c1482\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01640f00-7pfvq"
Nov 03 09:16:20.283493 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:20.283461    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01640f00-7pfvq"
Nov 03 09:16:20.329157 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:20.329124    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:16:20.702976 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:20.702924    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01640f00-7pfvq"]
Nov 03 09:16:20.703525 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:16:20.703493    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod212f6063_6386_4d83_9da1_3030eb8c1482.slice/crio-d6d08422d595212c226e7d1ebaa1e4942f6e4b0f618b0c65a795576959313446 WatchSource:0}: Error finding container d6d08422d595212c226e7d1ebaa1e4942f6e4b0f618b0c65a795576959313446: Status 404 returned error can't find the container with id d6d08422d595212c226e7d1ebaa1e4942f6e4b0f618b0c65a795576959313446
Nov 03 09:16:21.206844 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:21.206795    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01640f00-7pfvq" event={"ID":"212f6063-6386-4d83-9da1-3030eb8c1482","Type":"ContainerStarted","Data":"b9e74aa2ab6b0ee6bd5b1e1c63eebdffcec7d526597ac816112f58132d9491f3"}
Nov 03 09:16:21.207282 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:21.206886    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01640f00-7pfvq" event={"ID":"212f6063-6386-4d83-9da1-3030eb8c1482","Type":"ContainerStarted","Data":"d6d08422d595212c226e7d1ebaa1e4942f6e4b0f618b0c65a795576959313446"}
Nov 03 09:16:21.338721 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:21.338610    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01640f00-7pfvq" podStartSLOduration=2.338595149 podStartE2EDuration="2.338595149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:16:21.337490614 +0000 UTC m=+2205.606554049" watchObservedRunningTime="2026-11-03 09:16:21.338595149 +0000 UTC m=+2205.607658583"
Nov 03 09:16:22.403595 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:22.403558    4473 generic.go:356] "Generic (PLEG): container finished" podID="212f6063-6386-4d83-9da1-3030eb8c1482" containerID="b9e74aa2ab6b0ee6bd5b1e1c63eebdffcec7d526597ac816112f58132d9491f3" exitCode=0
Nov 03 09:16:22.404239 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:22.404203    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01640f00-7pfvq" event={"ID":"212f6063-6386-4d83-9da1-3030eb8c1482","Type":"ContainerDied","Data":"b9e74aa2ab6b0ee6bd5b1e1c63eebdffcec7d526597ac816112f58132d9491f3"}
Nov 03 09:16:24.105320 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:24.105295    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01640f00-7pfvq"
Nov 03 09:16:24.201176 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:24.201088    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/212f6063-6386-4d83-9da1-3030eb8c1482-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/212f6063-6386-4d83-9da1-3030eb8c1482-maas-api-xks-ca\") pod \"212f6063-6386-4d83-9da1-3030eb8c1482\" (UID: \"212f6063-6386-4d83-9da1-3030eb8c1482\") "
Nov 03 09:16:24.201176 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:24.201146    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/212f6063-6386-4d83-9da1-3030eb8c1482-kube-api-access-bvdhl\" (UniqueName: \"kubernetes.io/projected/212f6063-6386-4d83-9da1-3030eb8c1482-kube-api-access-bvdhl\") pod \"212f6063-6386-4d83-9da1-3030eb8c1482\" (UID: \"212f6063-6386-4d83-9da1-3030eb8c1482\") "
Nov 03 09:16:24.201482 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:24.201256    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/212f6063-6386-4d83-9da1-3030eb8c1482-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/212f6063-6386-4d83-9da1-3030eb8c1482-maas-api-openshift-ca\") pod \"212f6063-6386-4d83-9da1-3030eb8c1482\" (UID: \"212f6063-6386-4d83-9da1-3030eb8c1482\") "
Nov 03 09:16:24.201648 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:24.201617    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/212f6063-6386-4d83-9da1-3030eb8c1482-maas-api-openshift-ca" pod "212f6063-6386-4d83-9da1-3030eb8c1482" (UID: "212f6063-6386-4d83-9da1-3030eb8c1482"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:16:24.203422 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:24.203395    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/212f6063-6386-4d83-9da1-3030eb8c1482-kube-api-access-bvdhl" pod "212f6063-6386-4d83-9da1-3030eb8c1482" (UID: "212f6063-6386-4d83-9da1-3030eb8c1482"). InnerVolumeSpecName "kube-api-access-bvdhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:16:24.203525 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:24.203432    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/212f6063-6386-4d83-9da1-3030eb8c1482-maas-api-xks-ca" pod "212f6063-6386-4d83-9da1-3030eb8c1482" (UID: "212f6063-6386-4d83-9da1-3030eb8c1482"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:16:24.302473 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:24.302440    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/212f6063-6386-4d83-9da1-3030eb8c1482-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:16:24.302473 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:24.302469    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvdhl\" (UniqueName: \"kubernetes.io/projected/212f6063-6386-4d83-9da1-3030eb8c1482-kube-api-access-bvdhl\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:16:24.302473 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:24.302482    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/212f6063-6386-4d83-9da1-3030eb8c1482-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:16:24.426915 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:24.426880    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01640f00-7pfvq"
Nov 03 09:16:24.427139 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:24.426911    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01640f00-7pfvq" event={"ID":"212f6063-6386-4d83-9da1-3030eb8c1482","Type":"ContainerDied","Data":"d6d08422d595212c226e7d1ebaa1e4942f6e4b0f618b0c65a795576959313446"}
Nov 03 09:16:24.427139 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:24.426983    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6d08422d595212c226e7d1ebaa1e4942f6e4b0f618b0c65a795576959313446"
Nov 03 09:16:40.702164 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:40.702031    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx" podUID="8e84a474-def6-41dc-8516-5e5ccf42ff46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:16:40.990789 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:40.990076    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd" podUID="cf4cf2fc-612a-4f5c-a596-543036dde7e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:16:40.991686 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:40.991654    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh" podUID="00bab831-c667-415d-b6b0-81fff4d57762" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:16:43.088062 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:43.088028    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06de7aa4-kh7ng"]
Nov 03 09:16:43.088613 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:43.088598    4473 cpu_manager.go:414] "containerMap: removing container" podUID="212f6063-6386-4d83-9da1-3030eb8c1482" containerName="revoke-keys"
Nov 03 09:16:43.088656 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:43.088614    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="212f6063-6386-4d83-9da1-3030eb8c1482" containerName="revoke-keys"
Nov 03 09:16:43.088715 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:43.088706    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="212f6063-6386-4d83-9da1-3030eb8c1482" containerName="revoke-keys"
Nov 03 09:16:43.093266 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:43.093248    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06de7aa4-kh7ng"
Nov 03 09:16:43.149634 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:43.149604    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06de7aa4-kh7ng"]
Nov 03 09:16:43.165019 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:43.164982    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a250127-84fa-4808-9a98-abd9dd9afe63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06de7aa4-kh7ng\" (UID: \"9a250127-84fa-4808-9a98-abd9dd9afe63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06de7aa4-kh7ng"
Nov 03 09:16:43.165260 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:43.165230    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a250127-84fa-4808-9a98-abd9dd9afe63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06de7aa4-kh7ng\" (UID: \"9a250127-84fa-4808-9a98-abd9dd9afe63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06de7aa4-kh7ng"
Nov 03 09:16:43.165417 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:43.165351    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhkld\" (UniqueName: \"kubernetes.io/projected/9a250127-84fa-4808-9a98-abd9dd9afe63-kube-api-access-zhkld\") pod \"maas-api-revoke-sub-06de7aa4-kh7ng\" (UID: \"9a250127-84fa-4808-9a98-abd9dd9afe63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06de7aa4-kh7ng"
Nov 03 09:16:43.195778 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:43.195752    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:16:43.195919 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:43.195790    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:16:43.195919 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:43.195794    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:16:43.196824 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:43.196805    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:16:43.266808 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:43.266774    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a250127-84fa-4808-9a98-abd9dd9afe63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06de7aa4-kh7ng\" (UID: \"9a250127-84fa-4808-9a98-abd9dd9afe63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06de7aa4-kh7ng"
Nov 03 09:16:43.267002 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:43.266824    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhkld\" (UniqueName: \"kubernetes.io/projected/9a250127-84fa-4808-9a98-abd9dd9afe63-kube-api-access-zhkld\") pod \"maas-api-revoke-sub-06de7aa4-kh7ng\" (UID: \"9a250127-84fa-4808-9a98-abd9dd9afe63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06de7aa4-kh7ng"
Nov 03 09:16:43.267002 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:43.266855    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a250127-84fa-4808-9a98-abd9dd9afe63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06de7aa4-kh7ng\" (UID: \"9a250127-84fa-4808-9a98-abd9dd9afe63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06de7aa4-kh7ng"
Nov 03 09:16:43.267918 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:43.267891    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a250127-84fa-4808-9a98-abd9dd9afe63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06de7aa4-kh7ng\" (UID: \"9a250127-84fa-4808-9a98-abd9dd9afe63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06de7aa4-kh7ng"
Nov 03 09:16:43.269413 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:43.269392    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a250127-84fa-4808-9a98-abd9dd9afe63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06de7aa4-kh7ng\" (UID: \"9a250127-84fa-4808-9a98-abd9dd9afe63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06de7aa4-kh7ng"
Nov 03 09:16:43.326821 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:43.326790    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhkld\" (UniqueName: \"kubernetes.io/projected/9a250127-84fa-4808-9a98-abd9dd9afe63-kube-api-access-zhkld\") pod \"maas-api-revoke-sub-06de7aa4-kh7ng\" (UID: \"9a250127-84fa-4808-9a98-abd9dd9afe63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06de7aa4-kh7ng"
Nov 03 09:16:43.407056 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:43.407017    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06de7aa4-kh7ng"
Nov 03 09:16:43.787566 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:43.787542    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06de7aa4-kh7ng"]
Nov 03 09:16:43.788112 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:16:43.788084    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a250127_84fa_4808_9a98_abd9dd9afe63.slice/crio-9b8a79e66d305ae89637105ca77e9dd92a41cd2826d3bcd350a7f5faa79cccb6 WatchSource:0}: Error finding container 9b8a79e66d305ae89637105ca77e9dd92a41cd2826d3bcd350a7f5faa79cccb6: Status 404 returned error can't find the container with id 9b8a79e66d305ae89637105ca77e9dd92a41cd2826d3bcd350a7f5faa79cccb6
Nov 03 09:16:44.713326 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:44.713290    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06de7aa4-kh7ng" event={"ID":"9a250127-84fa-4808-9a98-abd9dd9afe63","Type":"ContainerStarted","Data":"32b84428177285af1433941a8200b8e139a3431797c5ce2492d1bea86193e4f9"}
Nov 03 09:16:44.713741 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:44.713386    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06de7aa4-kh7ng" event={"ID":"9a250127-84fa-4808-9a98-abd9dd9afe63","Type":"ContainerStarted","Data":"9b8a79e66d305ae89637105ca77e9dd92a41cd2826d3bcd350a7f5faa79cccb6"}
Nov 03 09:16:44.833761 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:44.833704    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06de7aa4-kh7ng" podStartSLOduration=1.833683899 podStartE2EDuration="1.833683899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:16:44.831293983 +0000 UTC m=+2229.100357440" watchObservedRunningTime="2026-11-03 09:16:44.833683899 +0000 UTC m=+2229.102747334"
Nov 03 09:16:45.720914 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:45.720875    4473 generic.go:356] "Generic (PLEG): container finished" podID="9a250127-84fa-4808-9a98-abd9dd9afe63" containerID="32b84428177285af1433941a8200b8e139a3431797c5ce2492d1bea86193e4f9" exitCode=0
Nov 03 09:16:45.721359 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:45.720988    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06de7aa4-kh7ng" event={"ID":"9a250127-84fa-4808-9a98-abd9dd9afe63","Type":"ContainerDied","Data":"32b84428177285af1433941a8200b8e139a3431797c5ce2492d1bea86193e4f9"}
Nov 03 09:16:46.877755 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:46.877731    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06de7aa4-kh7ng"
Nov 03 09:16:46.908214 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:46.908183    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a250127-84fa-4808-9a98-abd9dd9afe63-kube-api-access-zhkld\" (UniqueName: \"kubernetes.io/projected/9a250127-84fa-4808-9a98-abd9dd9afe63-kube-api-access-zhkld\") pod \"9a250127-84fa-4808-9a98-abd9dd9afe63\" (UID: \"9a250127-84fa-4808-9a98-abd9dd9afe63\") "
Nov 03 09:16:46.908380 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:46.908289    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a250127-84fa-4808-9a98-abd9dd9afe63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a250127-84fa-4808-9a98-abd9dd9afe63-maas-api-openshift-ca\") pod \"9a250127-84fa-4808-9a98-abd9dd9afe63\" (UID: \"9a250127-84fa-4808-9a98-abd9dd9afe63\") "
Nov 03 09:16:46.908380 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:46.908329    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a250127-84fa-4808-9a98-abd9dd9afe63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a250127-84fa-4808-9a98-abd9dd9afe63-maas-api-xks-ca\") pod \"9a250127-84fa-4808-9a98-abd9dd9afe63\" (UID: \"9a250127-84fa-4808-9a98-abd9dd9afe63\") "
Nov 03 09:16:46.908747 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:46.908717    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a250127-84fa-4808-9a98-abd9dd9afe63-maas-api-openshift-ca" pod "9a250127-84fa-4808-9a98-abd9dd9afe63" (UID: "9a250127-84fa-4808-9a98-abd9dd9afe63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:16:46.911014 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:46.910990    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a250127-84fa-4808-9a98-abd9dd9afe63-maas-api-xks-ca" pod "9a250127-84fa-4808-9a98-abd9dd9afe63" (UID: "9a250127-84fa-4808-9a98-abd9dd9afe63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:16:46.911171 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:46.911025    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a250127-84fa-4808-9a98-abd9dd9afe63-kube-api-access-zhkld" pod "9a250127-84fa-4808-9a98-abd9dd9afe63" (UID: "9a250127-84fa-4808-9a98-abd9dd9afe63"). InnerVolumeSpecName "kube-api-access-zhkld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:16:46.941020 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:16:46.940981    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:16:47.009817 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:47.009749    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhkld\" (UniqueName: \"kubernetes.io/projected/9a250127-84fa-4808-9a98-abd9dd9afe63-kube-api-access-zhkld\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:16:47.009817 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:47.009776    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a250127-84fa-4808-9a98-abd9dd9afe63-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:16:47.009817 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:47.009786    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a250127-84fa-4808-9a98-abd9dd9afe63-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:16:47.734577 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:47.734546    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06de7aa4-kh7ng"
Nov 03 09:16:47.734874 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:47.734589    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06de7aa4-kh7ng" event={"ID":"9a250127-84fa-4808-9a98-abd9dd9afe63","Type":"ContainerDied","Data":"9b8a79e66d305ae89637105ca77e9dd92a41cd2826d3bcd350a7f5faa79cccb6"}
Nov 03 09:16:47.734874 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:47.734645    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b8a79e66d305ae89637105ca77e9dd92a41cd2826d3bcd350a7f5faa79cccb6"
Nov 03 09:16:48.837961 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:48.837910    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c954621c-kxqb2"]
Nov 03 09:16:48.838441 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:48.838417    4473 cpu_manager.go:414] "containerMap: removing container" podUID="9a250127-84fa-4808-9a98-abd9dd9afe63" containerName="revoke-keys"
Nov 03 09:16:48.838441 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:48.838430    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a250127-84fa-4808-9a98-abd9dd9afe63" containerName="revoke-keys"
Nov 03 09:16:48.838564 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:48.838554    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a250127-84fa-4808-9a98-abd9dd9afe63" containerName="revoke-keys"
Nov 03 09:16:48.842073 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:48.842056    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c954621c-kxqb2"
Nov 03 09:16:48.899580 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:48.899548    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c954621c-kxqb2"]
Nov 03 09:16:48.932359 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:48.932317    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-958c2\" (UniqueName: \"kubernetes.io/projected/88a1b3ae-3cdd-43e9-b498-34ef27a3ead7-kube-api-access-958c2\") pod \"maas-api-revoke-sub-c954621c-kxqb2\" (UID: \"88a1b3ae-3cdd-43e9-b498-34ef27a3ead7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c954621c-kxqb2"
Nov 03 09:16:48.932546 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:48.932412    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88a1b3ae-3cdd-43e9-b498-34ef27a3ead7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c954621c-kxqb2\" (UID: \"88a1b3ae-3cdd-43e9-b498-34ef27a3ead7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c954621c-kxqb2"
Nov 03 09:16:48.932616 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:48.932594    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88a1b3ae-3cdd-43e9-b498-34ef27a3ead7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c954621c-kxqb2\" (UID: \"88a1b3ae-3cdd-43e9-b498-34ef27a3ead7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c954621c-kxqb2"
Nov 03 09:16:48.945122 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:48.945090    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:16:48.945122 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:48.945100    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:16:48.945317 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:48.945130    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:16:48.945355 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:48.945325    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:16:49.033377 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:49.033348    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-958c2\" (UniqueName: \"kubernetes.io/projected/88a1b3ae-3cdd-43e9-b498-34ef27a3ead7-kube-api-access-958c2\") pod \"maas-api-revoke-sub-c954621c-kxqb2\" (UID: \"88a1b3ae-3cdd-43e9-b498-34ef27a3ead7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c954621c-kxqb2"
Nov 03 09:16:49.033540 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:49.033398    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88a1b3ae-3cdd-43e9-b498-34ef27a3ead7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c954621c-kxqb2\" (UID: \"88a1b3ae-3cdd-43e9-b498-34ef27a3ead7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c954621c-kxqb2"
Nov 03 09:16:49.033540 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:49.033437    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88a1b3ae-3cdd-43e9-b498-34ef27a3ead7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c954621c-kxqb2\" (UID: \"88a1b3ae-3cdd-43e9-b498-34ef27a3ead7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c954621c-kxqb2"
Nov 03 09:16:49.034243 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:49.034217    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88a1b3ae-3cdd-43e9-b498-34ef27a3ead7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c954621c-kxqb2\" (UID: \"88a1b3ae-3cdd-43e9-b498-34ef27a3ead7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c954621c-kxqb2"
Nov 03 09:16:49.035889 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:49.035865    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88a1b3ae-3cdd-43e9-b498-34ef27a3ead7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c954621c-kxqb2\" (UID: \"88a1b3ae-3cdd-43e9-b498-34ef27a3ead7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c954621c-kxqb2"
Nov 03 09:16:49.094686 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:49.094618    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-958c2\" (UniqueName: \"kubernetes.io/projected/88a1b3ae-3cdd-43e9-b498-34ef27a3ead7-kube-api-access-958c2\") pod \"maas-api-revoke-sub-c954621c-kxqb2\" (UID: \"88a1b3ae-3cdd-43e9-b498-34ef27a3ead7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c954621c-kxqb2"
Nov 03 09:16:49.156640 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:49.156613    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c954621c-kxqb2"
Nov 03 09:16:49.534688 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:16:49.534647    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88a1b3ae_3cdd_43e9_b498_34ef27a3ead7.slice/crio-def72301d0a7fd598a2411d790616dadee1971cf8e619f105accb5b680d999bb WatchSource:0}: Error finding container def72301d0a7fd598a2411d790616dadee1971cf8e619f105accb5b680d999bb: Status 404 returned error can't find the container with id def72301d0a7fd598a2411d790616dadee1971cf8e619f105accb5b680d999bb
Nov 03 09:16:49.535001 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:49.534979    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c954621c-kxqb2"]
Nov 03 09:16:49.817044 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:49.815597    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c954621c-kxqb2" event={"ID":"88a1b3ae-3cdd-43e9-b498-34ef27a3ead7","Type":"ContainerStarted","Data":"def72301d0a7fd598a2411d790616dadee1971cf8e619f105accb5b680d999bb"}
Nov 03 09:16:50.269395 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:50.269360    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-121ee6-6b46c6847b-qsf5t"]
Nov 03 09:16:50.269815 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:50.269553    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-121ee6-6b46c6847b-qsf5t" podUID="9c47f7a0-c0b1-4076-b122-ff49eb32fb60" containerName="payload-processing" containerID="cri-o://d32dc9ff519012d63a8c42753a1036789bd4266cc415b40e2b2fcae9bf6abdf2" gracePeriod=30
Nov 03 09:16:50.321085 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:50.321055    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-8694bd86f9-4vb49"]
Nov 03 09:16:50.321264 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:50.321245    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-8694bd86f9-4vb49" podUID="92cc6b00-3639-4fdf-beab-510019250b68" containerName="payload-pre-processing" containerID="cri-o://197b48e27a56f22f853e8895a41616856c4ac7a93e189e09eb89aa9017b69408" gracePeriod=30
Nov 03 09:16:50.640109 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:50.640085    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-121ee6-6b46c6847b-qsf5t"
Nov 03 09:16:50.643287 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:50.643262    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-8694bd86f9-4vb49"
Nov 03 09:16:50.669678 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:50.669647    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92cc6b00-3639-4fdf-beab-510019250b68-kube-api-access-wvmkl\" (UniqueName: \"kubernetes.io/projected/92cc6b00-3639-4fdf-beab-510019250b68-kube-api-access-wvmkl\") pod \"92cc6b00-3639-4fdf-beab-510019250b68\" (UID: \"92cc6b00-3639-4fdf-beab-510019250b68\") "
Nov 03 09:16:50.669856 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:50.669757    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c47f7a0-c0b1-4076-b122-ff49eb32fb60-kube-api-access-bplfn\" (UniqueName: \"kubernetes.io/projected/9c47f7a0-c0b1-4076-b122-ff49eb32fb60-kube-api-access-bplfn\") pod \"9c47f7a0-c0b1-4076-b122-ff49eb32fb60\" (UID: \"9c47f7a0-c0b1-4076-b122-ff49eb32fb60\") "
Nov 03 09:16:50.669856 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:50.669838    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c47f7a0-c0b1-4076-b122-ff49eb32fb60-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c47f7a0-c0b1-4076-b122-ff49eb32fb60-plugins-config-volume\") pod \"9c47f7a0-c0b1-4076-b122-ff49eb32fb60\" (UID: \"9c47f7a0-c0b1-4076-b122-ff49eb32fb60\") "
Nov 03 09:16:50.669971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:50.669882    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92cc6b00-3639-4fdf-beab-510019250b68-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92cc6b00-3639-4fdf-beab-510019250b68-plugins-config-volume\") pod \"92cc6b00-3639-4fdf-beab-510019250b68\" (UID: \"92cc6b00-3639-4fdf-beab-510019250b68\") "
Nov 03 09:16:50.670279 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:50.670253    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c47f7a0-c0b1-4076-b122-ff49eb32fb60-plugins-config-volume" pod "9c47f7a0-c0b1-4076-b122-ff49eb32fb60" (UID: "9c47f7a0-c0b1-4076-b122-ff49eb32fb60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:16:50.670279 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:50.670266    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92cc6b00-3639-4fdf-beab-510019250b68-plugins-config-volume" pod "92cc6b00-3639-4fdf-beab-510019250b68" (UID: "92cc6b00-3639-4fdf-beab-510019250b68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:16:50.670449 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:50.670433    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c47f7a0-c0b1-4076-b122-ff49eb32fb60-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:16:50.670449 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:50.670446    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92cc6b00-3639-4fdf-beab-510019250b68-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:16:50.672654 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:50.672632    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c47f7a0-c0b1-4076-b122-ff49eb32fb60-kube-api-access-bplfn" pod "9c47f7a0-c0b1-4076-b122-ff49eb32fb60" (UID: "9c47f7a0-c0b1-4076-b122-ff49eb32fb60"). InnerVolumeSpecName "kube-api-access-bplfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:16:50.672755 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:50.672662    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92cc6b00-3639-4fdf-beab-510019250b68-kube-api-access-wvmkl" pod "92cc6b00-3639-4fdf-beab-510019250b68" (UID: "92cc6b00-3639-4fdf-beab-510019250b68"). InnerVolumeSpecName "kube-api-access-wvmkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:16:50.771595 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:50.771561    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bplfn\" (UniqueName: \"kubernetes.io/projected/9c47f7a0-c0b1-4076-b122-ff49eb32fb60-kube-api-access-bplfn\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:16:50.771595 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:50.771591    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvmkl\" (UniqueName: \"kubernetes.io/projected/92cc6b00-3639-4fdf-beab-510019250b68-kube-api-access-wvmkl\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:16:50.900972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:50.896742    4473 generic.go:356] "Generic (PLEG): container finished" podID="92cc6b00-3639-4fdf-beab-510019250b68" containerID="197b48e27a56f22f853e8895a41616856c4ac7a93e189e09eb89aa9017b69408" exitCode=0
Nov 03 09:16:50.900972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:50.896914    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-8694bd86f9-4vb49" event={"ID":"92cc6b00-3639-4fdf-beab-510019250b68","Type":"ContainerDied","Data":"197b48e27a56f22f853e8895a41616856c4ac7a93e189e09eb89aa9017b69408"}
Nov 03 09:16:50.900972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:50.896994    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-8694bd86f9-4vb49" event={"ID":"92cc6b00-3639-4fdf-beab-510019250b68","Type":"ContainerDied","Data":"80998db9d14aec720f2287944cef476f6447d20c6d67cd0a3323af75896b1f73"}
Nov 03 09:16:50.900972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:50.897030    4473 scope.go:122] "RemoveContainer" containerID="197b48e27a56f22f853e8895a41616856c4ac7a93e189e09eb89aa9017b69408"
Nov 03 09:16:50.900972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:50.897838    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-8694bd86f9-4vb49"
Nov 03 09:16:50.912989 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:50.909700    4473 generic.go:356] "Generic (PLEG): container finished" podID="9c47f7a0-c0b1-4076-b122-ff49eb32fb60" containerID="d32dc9ff519012d63a8c42753a1036789bd4266cc415b40e2b2fcae9bf6abdf2" exitCode=0
Nov 03 09:16:50.912989 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:50.909810    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-121ee6-6b46c6847b-qsf5t" event={"ID":"9c47f7a0-c0b1-4076-b122-ff49eb32fb60","Type":"ContainerDied","Data":"d32dc9ff519012d63a8c42753a1036789bd4266cc415b40e2b2fcae9bf6abdf2"}
Nov 03 09:16:50.912989 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:50.909859    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-121ee6-6b46c6847b-qsf5t" event={"ID":"9c47f7a0-c0b1-4076-b122-ff49eb32fb60","Type":"ContainerDied","Data":"6d7473c415c11d0404f81dce133ece7fc8479291d2206dc16bbdba2c99044845"}
Nov 03 09:16:50.912989 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:50.909927    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-121ee6-6b46c6847b-qsf5t"
Nov 03 09:16:50.912989 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:50.912796    4473 generic.go:356] "Generic (PLEG): container finished" podID="88a1b3ae-3cdd-43e9-b498-34ef27a3ead7" containerID="a5ee69b7f94061caff00b09744d65fbc3beb8246a7627a72c9490cbce605b709" exitCode=0
Nov 03 09:16:50.912989 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:50.912862    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c954621c-kxqb2" event={"ID":"88a1b3ae-3cdd-43e9-b498-34ef27a3ead7","Type":"ContainerDied","Data":"a5ee69b7f94061caff00b09744d65fbc3beb8246a7627a72c9490cbce605b709"}
Nov 03 09:16:50.958741 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:50.958715    4473 scope.go:122] "RemoveContainer" containerID="197b48e27a56f22f853e8895a41616856c4ac7a93e189e09eb89aa9017b69408"
Nov 03 09:16:50.959205 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:16:50.959179    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"197b48e27a56f22f853e8895a41616856c4ac7a93e189e09eb89aa9017b69408\": container with ID starting with 197b48e27a56f22f853e8895a41616856c4ac7a93e189e09eb89aa9017b69408 not found: ID does not exist" containerID="197b48e27a56f22f853e8895a41616856c4ac7a93e189e09eb89aa9017b69408"
Nov 03 09:16:50.959314 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:50.959211    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"197b48e27a56f22f853e8895a41616856c4ac7a93e189e09eb89aa9017b69408"} err="failed to get container status \"197b48e27a56f22f853e8895a41616856c4ac7a93e189e09eb89aa9017b69408\": rpc error: code = NotFound desc = could not find container \"197b48e27a56f22f853e8895a41616856c4ac7a93e189e09eb89aa9017b69408\": container with ID starting with 197b48e27a56f22f853e8895a41616856c4ac7a93e189e09eb89aa9017b69408 not found: ID does not exist"
Nov 03 09:16:50.959314 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:50.959227    4473 scope.go:122] "RemoveContainer" containerID="d32dc9ff519012d63a8c42753a1036789bd4266cc415b40e2b2fcae9bf6abdf2"
Nov 03 09:16:50.970284 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:50.970258    4473 scope.go:122] "RemoveContainer" containerID="d32dc9ff519012d63a8c42753a1036789bd4266cc415b40e2b2fcae9bf6abdf2"
Nov 03 09:16:50.970632 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:16:50.970606    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d32dc9ff519012d63a8c42753a1036789bd4266cc415b40e2b2fcae9bf6abdf2\": container with ID starting with d32dc9ff519012d63a8c42753a1036789bd4266cc415b40e2b2fcae9bf6abdf2 not found: ID does not exist" containerID="d32dc9ff519012d63a8c42753a1036789bd4266cc415b40e2b2fcae9bf6abdf2"
Nov 03 09:16:50.970758 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:50.970636    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d32dc9ff519012d63a8c42753a1036789bd4266cc415b40e2b2fcae9bf6abdf2"} err="failed to get container status \"d32dc9ff519012d63a8c42753a1036789bd4266cc415b40e2b2fcae9bf6abdf2\": rpc error: code = NotFound desc = could not find container \"d32dc9ff519012d63a8c42753a1036789bd4266cc415b40e2b2fcae9bf6abdf2\": container with ID starting with d32dc9ff519012d63a8c42753a1036789bd4266cc415b40e2b2fcae9bf6abdf2 not found: ID does not exist"
Nov 03 09:16:51.075903 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:51.075865    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-121ee6-6b46c6847b-qsf5t"]
Nov 03 09:16:51.080514 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:51.080489    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-121ee6-6b46c6847b-qsf5t"]
Nov 03 09:16:51.309354 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:51.309265    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-8694bd86f9-4vb49"]
Nov 03 09:16:51.315070 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:51.315033    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-121ee6-8694bd86f9-4vb49"]
Nov 03 09:16:51.871143 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:51.871104    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92cc6b00-3639-4fdf-beab-510019250b68" path="/var/lib/kubelet/pods/92cc6b00-3639-4fdf-beab-510019250b68/volumes"
Nov 03 09:16:51.871550 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:51.871530    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c47f7a0-c0b1-4076-b122-ff49eb32fb60" path="/var/lib/kubelet/pods/9c47f7a0-c0b1-4076-b122-ff49eb32fb60/volumes"
Nov 03 09:16:52.102029 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:52.102004    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c954621c-kxqb2"
Nov 03 09:16:52.190591 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:52.190522    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88a1b3ae-3cdd-43e9-b498-34ef27a3ead7-kube-api-access-958c2\" (UniqueName: \"kubernetes.io/projected/88a1b3ae-3cdd-43e9-b498-34ef27a3ead7-kube-api-access-958c2\") pod \"88a1b3ae-3cdd-43e9-b498-34ef27a3ead7\" (UID: \"88a1b3ae-3cdd-43e9-b498-34ef27a3ead7\") "
Nov 03 09:16:52.190729 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:52.190695    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88a1b3ae-3cdd-43e9-b498-34ef27a3ead7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88a1b3ae-3cdd-43e9-b498-34ef27a3ead7-maas-api-xks-ca\") pod \"88a1b3ae-3cdd-43e9-b498-34ef27a3ead7\" (UID: \"88a1b3ae-3cdd-43e9-b498-34ef27a3ead7\") "
Nov 03 09:16:52.190729 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:52.190718    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88a1b3ae-3cdd-43e9-b498-34ef27a3ead7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88a1b3ae-3cdd-43e9-b498-34ef27a3ead7-maas-api-openshift-ca\") pod \"88a1b3ae-3cdd-43e9-b498-34ef27a3ead7\" (UID: \"88a1b3ae-3cdd-43e9-b498-34ef27a3ead7\") "
Nov 03 09:16:52.191132 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:52.191107    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88a1b3ae-3cdd-43e9-b498-34ef27a3ead7-maas-api-openshift-ca" pod "88a1b3ae-3cdd-43e9-b498-34ef27a3ead7" (UID: "88a1b3ae-3cdd-43e9-b498-34ef27a3ead7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:16:52.192837 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:52.192813    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88a1b3ae-3cdd-43e9-b498-34ef27a3ead7-kube-api-access-958c2" pod "88a1b3ae-3cdd-43e9-b498-34ef27a3ead7" (UID: "88a1b3ae-3cdd-43e9-b498-34ef27a3ead7"). InnerVolumeSpecName "kube-api-access-958c2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:16:52.193016 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:52.192968    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88a1b3ae-3cdd-43e9-b498-34ef27a3ead7-maas-api-xks-ca" pod "88a1b3ae-3cdd-43e9-b498-34ef27a3ead7" (UID: "88a1b3ae-3cdd-43e9-b498-34ef27a3ead7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:16:52.292352 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:52.292320    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-958c2\" (UniqueName: \"kubernetes.io/projected/88a1b3ae-3cdd-43e9-b498-34ef27a3ead7-kube-api-access-958c2\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:16:52.292352 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:52.292349    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88a1b3ae-3cdd-43e9-b498-34ef27a3ead7-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:16:52.292352 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:52.292358    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88a1b3ae-3cdd-43e9-b498-34ef27a3ead7-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:16:52.933904 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:52.933867    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c954621c-kxqb2" event={"ID":"88a1b3ae-3cdd-43e9-b498-34ef27a3ead7","Type":"ContainerDied","Data":"def72301d0a7fd598a2411d790616dadee1971cf8e619f105accb5b680d999bb"}
Nov 03 09:16:52.934348 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:52.933927    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="def72301d0a7fd598a2411d790616dadee1971cf8e619f105accb5b680d999bb"
Nov 03 09:16:52.934348 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:52.933879    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c954621c-kxqb2"
Nov 03 09:16:53.880997 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:53.880969    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"]
Nov 03 09:16:53.881461 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:53.881447    4473 cpu_manager.go:414] "containerMap: removing container" podUID="92cc6b00-3639-4fdf-beab-510019250b68" containerName="payload-pre-processing"
Nov 03 09:16:53.881506 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:53.881463    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92cc6b00-3639-4fdf-beab-510019250b68" containerName="payload-pre-processing"
Nov 03 09:16:53.881506 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:53.881485    4473 cpu_manager.go:414] "containerMap: removing container" podUID="88a1b3ae-3cdd-43e9-b498-34ef27a3ead7" containerName="revoke-keys"
Nov 03 09:16:53.881506 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:53.881495    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88a1b3ae-3cdd-43e9-b498-34ef27a3ead7" containerName="revoke-keys"
Nov 03 09:16:53.881604 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:53.881522    4473 cpu_manager.go:414] "containerMap: removing container" podUID="9c47f7a0-c0b1-4076-b122-ff49eb32fb60" containerName="payload-processing"
Nov 03 09:16:53.881604 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:53.881527    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c47f7a0-c0b1-4076-b122-ff49eb32fb60" containerName="payload-processing"
Nov 03 09:16:53.881678 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:53.881608    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c47f7a0-c0b1-4076-b122-ff49eb32fb60" containerName="payload-processing"
Nov 03 09:16:53.881678 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:53.881619    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="92cc6b00-3639-4fdf-beab-510019250b68" containerName="payload-pre-processing"
Nov 03 09:16:53.881678 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:53.881626    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="88a1b3ae-3cdd-43e9-b498-34ef27a3ead7" containerName="revoke-keys"
Nov 03 09:16:53.885200 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:53.885177    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"
Nov 03 09:16:53.991971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:53.991916    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"]
Nov 03 09:16:54.013268 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:54.013233    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpf9q\" (UniqueName: \"kubernetes.io/projected/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-kube-api-access-vpf9q\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns\" (UID: \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"
Nov 03 09:16:54.013442 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:54.013285    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns\" (UID: \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"
Nov 03 09:16:54.013492 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:54.013447    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns\" (UID: \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"
Nov 03 09:16:54.013543 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:54.013490    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns\" (UID: \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"
Nov 03 09:16:54.014070 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:54.013582    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns\" (UID: \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"
Nov 03 09:16:54.014070 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:54.013703    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns\" (UID: \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"
Nov 03 09:16:54.014070 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:54.013750    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns\" (UID: \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"
Nov 03 09:16:54.014070 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:54.013874    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns\" (UID: \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"
Nov 03 09:16:54.014070 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:54.013925    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns\" (UID: \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"
Nov 03 09:16:54.114769 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:54.114741    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns\" (UID: \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"
Nov 03 09:16:54.114956 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:54.114804    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns\" (UID: \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"
Nov 03 09:16:54.114956 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:54.114828    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns\" (UID: \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"
Nov 03 09:16:54.114956 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:54.114877    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpf9q\" (UniqueName: \"kubernetes.io/projected/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-kube-api-access-vpf9q\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns\" (UID: \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"
Nov 03 09:16:54.114956 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:54.114898    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns\" (UID: \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"
Nov 03 09:16:54.115184 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:54.114985    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns\" (UID: \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"
Nov 03 09:16:54.115184 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:54.115020    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns\" (UID: \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"
Nov 03 09:16:54.115184 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:54.115079    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns\" (UID: \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"
Nov 03 09:16:54.115184 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:54.115152    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns\" (UID: \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"
Nov 03 09:16:54.115383 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:54.115261    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-workload-certs\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns\" (UID: \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"
Nov 03 09:16:54.115424 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:54.115403    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-credential-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns\" (UID: \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"
Nov 03 09:16:54.115489 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:54.115461    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-istio-data\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns\" (UID: \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"
Nov 03 09:16:54.115567 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:54.115524    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-istiod-ca-cert\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns\" (UID: \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"
Nov 03 09:16:54.115673 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:54.115647    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-workload-socket\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns\" (UID: \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"
Nov 03 09:16:54.117480 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:54.117457    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-istio-envoy\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns\" (UID: \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"
Nov 03 09:16:54.117617 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:54.117599    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-istio-podinfo\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns\" (UID: \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"
Nov 03 09:16:54.177633 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:54.177563    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-istio-token\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns\" (UID: \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"
Nov 03 09:16:54.187591 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:54.187566    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpf9q\" (UniqueName: \"kubernetes.io/projected/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-kube-api-access-vpf9q\") pod \"e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns\" (UID: \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\") " pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"
Nov 03 09:16:54.198568 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:54.198545    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"
Nov 03 09:16:54.630250 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:16:54.630216    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6da2e774_1fbc_445e_88f2_2e08a2b3b7e3.slice/crio-d52fdaaeaecefcbf3cc4fc7d312d1a536e842f5d3ffc131e9fc53f600ed1599c WatchSource:0}: Error finding container d52fdaaeaecefcbf3cc4fc7d312d1a536e842f5d3ffc131e9fc53f600ed1599c: Status 404 returned error can't find the container with id d52fdaaeaecefcbf3cc4fc7d312d1a536e842f5d3ffc131e9fc53f600ed1599c
Nov 03 09:16:54.630749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:54.630722    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"]
Nov 03 09:16:54.632563 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:54.632532    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:16:54.632641 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:54.632594    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:16:54.632641 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:54.632622    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:16:54.949422 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:54.949319    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns" event={"ID":"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3","Type":"ContainerStarted","Data":"a5abe396d6eaa91ccd0702015c9207889997f2d67883a9da10fda1fc1b82ab33"}
Nov 03 09:16:54.949422 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:54.949385    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns" event={"ID":"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3","Type":"ContainerStarted","Data":"d52fdaaeaecefcbf3cc4fc7d312d1a536e842f5d3ffc131e9fc53f600ed1599c"}
Nov 03 09:16:55.070915 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:55.070849    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns" podStartSLOduration=2.070831619 podStartE2EDuration="2.070831619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:16:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:16:55.069014159 +0000 UTC m=+2239.338077619" watchObservedRunningTime="2026-11-03 09:16:55.070831619 +0000 UTC m=+2239.339895056"
Nov 03 09:16:55.199172 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:55.199139    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"
Nov 03 09:16:55.200749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:55.200678    4473 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 03 09:16:55.200749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:55.200737    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns" podUID="6da2e774-1fbc-445e-88f2-2e08a2b3b7e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 03 09:16:56.199525 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:56.199489    4473 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 03 09:16:56.199937 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:56.199550    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns" podUID="6da2e774-1fbc-445e-88f2-2e08a2b3b7e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 03 09:16:57.199106 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:57.199073    4473 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 03 09:16:57.199283 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:57.199138    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns" podUID="6da2e774-1fbc-445e-88f2-2e08a2b3b7e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 03 09:16:57.870302 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:57.870276    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"]
Nov 03 09:16:57.870730 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:57.870445    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66" podUID="e9f837be-e9a7-4781-bb0d-8b7a50f63555" containerName="istio-proxy" containerID="cri-o://b61b53e74603e54735baf74ef1b15bb2d57990d04f0af046aef41a16a84bd753" gracePeriod=30
Nov 03 09:16:58.199083 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:58.198989    4473 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 03 09:16:58.199083 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:58.199046    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns" podUID="6da2e774-1fbc-445e-88f2-2e08a2b3b7e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 03 09:16:59.044738 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:59.044698    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fadbc327-trn6s"]
Nov 03 09:16:59.048829 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:59.048808    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fadbc327-trn6s"
Nov 03 09:16:59.082905 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:59.082873    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24ee886a-7918-4021-a818-0cdea41cf044-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fadbc327-trn6s\" (UID: \"24ee886a-7918-4021-a818-0cdea41cf044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fadbc327-trn6s"
Nov 03 09:16:59.083054 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:59.082995    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptljk\" (UniqueName: \"kubernetes.io/projected/24ee886a-7918-4021-a818-0cdea41cf044-kube-api-access-ptljk\") pod \"maas-api-revoke-sub-fadbc327-trn6s\" (UID: \"24ee886a-7918-4021-a818-0cdea41cf044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fadbc327-trn6s"
Nov 03 09:16:59.083054 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:59.083046    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24ee886a-7918-4021-a818-0cdea41cf044-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fadbc327-trn6s\" (UID: \"24ee886a-7918-4021-a818-0cdea41cf044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fadbc327-trn6s"
Nov 03 09:16:59.103398 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:59.103363    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fadbc327-trn6s"]
Nov 03 09:16:59.149727 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:59.149692    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:16:59.149896 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:59.149737    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:16:59.149896 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:59.149740    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:16:59.152577 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:59.152551    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:16:59.184036 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:59.183999    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24ee886a-7918-4021-a818-0cdea41cf044-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fadbc327-trn6s\" (UID: \"24ee886a-7918-4021-a818-0cdea41cf044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fadbc327-trn6s"
Nov 03 09:16:59.184227 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:59.184095    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptljk\" (UniqueName: \"kubernetes.io/projected/24ee886a-7918-4021-a818-0cdea41cf044-kube-api-access-ptljk\") pod \"maas-api-revoke-sub-fadbc327-trn6s\" (UID: \"24ee886a-7918-4021-a818-0cdea41cf044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fadbc327-trn6s"
Nov 03 09:16:59.184227 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:59.184141    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24ee886a-7918-4021-a818-0cdea41cf044-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fadbc327-trn6s\" (UID: \"24ee886a-7918-4021-a818-0cdea41cf044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fadbc327-trn6s"
Nov 03 09:16:59.184832 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:59.184813    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24ee886a-7918-4021-a818-0cdea41cf044-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fadbc327-trn6s\" (UID: \"24ee886a-7918-4021-a818-0cdea41cf044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fadbc327-trn6s"
Nov 03 09:16:59.186576 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:59.186555    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24ee886a-7918-4021-a818-0cdea41cf044-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fadbc327-trn6s\" (UID: \"24ee886a-7918-4021-a818-0cdea41cf044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fadbc327-trn6s"
Nov 03 09:16:59.199721 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:59.199695    4473 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 03 09:16:59.199830 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:59.199739    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns" podUID="6da2e774-1fbc-445e-88f2-2e08a2b3b7e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 03 09:16:59.244277 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:59.244239    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptljk\" (UniqueName: \"kubernetes.io/projected/24ee886a-7918-4021-a818-0cdea41cf044-kube-api-access-ptljk\") pod \"maas-api-revoke-sub-fadbc327-trn6s\" (UID: \"24ee886a-7918-4021-a818-0cdea41cf044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fadbc327-trn6s"
Nov 03 09:16:59.365013 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:59.362404    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fadbc327-trn6s"
Nov 03 09:16:59.777599 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:59.777563    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fadbc327-trn6s"]
Nov 03 09:16:59.777789 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:16:59.777763    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ee886a_7918_4021_a818_0cdea41cf044.slice/crio-44c2e5b3e258003676b77f8b1e280ca86395160683316c7d4a06d1c046a67f33 WatchSource:0}: Error finding container 44c2e5b3e258003676b77f8b1e280ca86395160683316c7d4a06d1c046a67f33: Status 404 returned error can't find the container with id 44c2e5b3e258003676b77f8b1e280ca86395160683316c7d4a06d1c046a67f33
Nov 03 09:16:59.982042 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:16:59.982005    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fadbc327-trn6s" event={"ID":"24ee886a-7918-4021-a818-0cdea41cf044","Type":"ContainerStarted","Data":"44c2e5b3e258003676b77f8b1e280ca86395160683316c7d4a06d1c046a67f33"}
Nov 03 09:17:00.199843 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:00.199813    4473 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 03 09:17:00.200208 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:00.199865    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns" podUID="6da2e774-1fbc-445e-88f2-2e08a2b3b7e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 03 09:17:00.990995 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:00.990886    4473 generic.go:356] "Generic (PLEG): container finished" podID="24ee886a-7918-4021-a818-0cdea41cf044" containerID="861bf4059b8a3ae883eeed366790a36f43a68d7b302a3edf1600ffdbb935fd77" exitCode=0
Nov 03 09:17:00.990995 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:00.990979    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fadbc327-trn6s" event={"ID":"24ee886a-7918-4021-a818-0cdea41cf044","Type":"ContainerDied","Data":"861bf4059b8a3ae883eeed366790a36f43a68d7b302a3edf1600ffdbb935fd77"}
Nov 03 09:17:01.203969 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:01.199475    4473 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body=
Nov 03 09:17:01.203969 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:01.199539    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns" podUID="6da2e774-1fbc-445e-88f2-2e08a2b3b7e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused"
Nov 03 09:17:02.145745 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:02.145721    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fadbc327-trn6s"
Nov 03 09:17:02.203341 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:02.203314    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"
Nov 03 09:17:02.203611 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:02.203592    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"
Nov 03 09:17:02.204188 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:02.204175    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"
Nov 03 09:17:02.225572 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:02.225547    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24ee886a-7918-4021-a818-0cdea41cf044-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24ee886a-7918-4021-a818-0cdea41cf044-maas-api-openshift-ca\") pod \"24ee886a-7918-4021-a818-0cdea41cf044\" (UID: \"24ee886a-7918-4021-a818-0cdea41cf044\") "
Nov 03 09:17:02.225700 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:02.225615    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24ee886a-7918-4021-a818-0cdea41cf044-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24ee886a-7918-4021-a818-0cdea41cf044-maas-api-xks-ca\") pod \"24ee886a-7918-4021-a818-0cdea41cf044\" (UID: \"24ee886a-7918-4021-a818-0cdea41cf044\") "
Nov 03 09:17:02.225738 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:02.225727    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24ee886a-7918-4021-a818-0cdea41cf044-kube-api-access-ptljk\" (UniqueName: \"kubernetes.io/projected/24ee886a-7918-4021-a818-0cdea41cf044-kube-api-access-ptljk\") pod \"24ee886a-7918-4021-a818-0cdea41cf044\" (UID: \"24ee886a-7918-4021-a818-0cdea41cf044\") "
Nov 03 09:17:02.226044 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:02.226005    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24ee886a-7918-4021-a818-0cdea41cf044-maas-api-openshift-ca" pod "24ee886a-7918-4021-a818-0cdea41cf044" (UID: "24ee886a-7918-4021-a818-0cdea41cf044"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:17:02.227996 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:02.227973    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ee886a-7918-4021-a818-0cdea41cf044-kube-api-access-ptljk" pod "24ee886a-7918-4021-a818-0cdea41cf044" (UID: "24ee886a-7918-4021-a818-0cdea41cf044"). InnerVolumeSpecName "kube-api-access-ptljk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:17:02.227996 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:02.227986    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24ee886a-7918-4021-a818-0cdea41cf044-maas-api-xks-ca" pod "24ee886a-7918-4021-a818-0cdea41cf044" (UID: "24ee886a-7918-4021-a818-0cdea41cf044"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:17:02.327253 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:02.327159    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptljk\" (UniqueName: \"kubernetes.io/projected/24ee886a-7918-4021-a818-0cdea41cf044-kube-api-access-ptljk\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:17:02.327253 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:02.327193    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24ee886a-7918-4021-a818-0cdea41cf044-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:17:02.327253 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:02.327212    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24ee886a-7918-4021-a818-0cdea41cf044-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:17:03.014431 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:03.014393    4473 generic.go:356] "Generic (PLEG): container finished" podID="e9f837be-e9a7-4781-bb0d-8b7a50f63555" containerID="b61b53e74603e54735baf74ef1b15bb2d57990d04f0af046aef41a16a84bd753" exitCode=0
Nov 03 09:17:03.014597 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:03.014465    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66" event={"ID":"e9f837be-e9a7-4781-bb0d-8b7a50f63555","Type":"ContainerDied","Data":"b61b53e74603e54735baf74ef1b15bb2d57990d04f0af046aef41a16a84bd753"}
Nov 03 09:17:03.017611 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:03.016837    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fadbc327-trn6s" event={"ID":"24ee886a-7918-4021-a818-0cdea41cf044","Type":"ContainerDied","Data":"44c2e5b3e258003676b77f8b1e280ca86395160683316c7d4a06d1c046a67f33"}
Nov 03 09:17:03.017611 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:03.016871    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fadbc327-trn6s"
Nov 03 09:17:03.017611 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:03.016895    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44c2e5b3e258003676b77f8b1e280ca86395160683316c7d4a06d1c046a67f33"
Nov 03 09:17:03.149393 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:03.149338    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"
Nov 03 09:17:03.244972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:03.244586    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9f837be-e9a7-4781-bb0d-8b7a50f63555-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9f837be-e9a7-4781-bb0d-8b7a50f63555-credential-socket\") pod \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\" (UID: \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\") "
Nov 03 09:17:03.244972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:03.244638    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9f837be-e9a7-4781-bb0d-8b7a50f63555-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9f837be-e9a7-4781-bb0d-8b7a50f63555-istiod-ca-cert\") pod \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\" (UID: \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\") "
Nov 03 09:17:03.244972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:03.244767    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9f837be-e9a7-4781-bb0d-8b7a50f63555-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9f837be-e9a7-4781-bb0d-8b7a50f63555-workload-socket\") pod \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\" (UID: \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\") "
Nov 03 09:17:03.244972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:03.244813    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9f837be-e9a7-4781-bb0d-8b7a50f63555-kube-api-access-dht5v\" (UniqueName: \"kubernetes.io/projected/e9f837be-e9a7-4781-bb0d-8b7a50f63555-kube-api-access-dht5v\") pod \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\" (UID: \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\") "
Nov 03 09:17:03.244972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:03.244855    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9f837be-e9a7-4781-bb0d-8b7a50f63555-istio-token\" (UniqueName: \"kubernetes.io/projected/e9f837be-e9a7-4781-bb0d-8b7a50f63555-istio-token\") pod \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\" (UID: \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\") "
Nov 03 09:17:03.245694 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:03.244978    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9f837be-e9a7-4781-bb0d-8b7a50f63555-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9f837be-e9a7-4781-bb0d-8b7a50f63555-istio-envoy\") pod \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\" (UID: \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\") "
Nov 03 09:17:03.245694 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:03.245109    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9f837be-e9a7-4781-bb0d-8b7a50f63555-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9f837be-e9a7-4781-bb0d-8b7a50f63555-istio-podinfo\") pod \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\" (UID: \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\") "
Nov 03 09:17:03.245694 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:03.245149    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9f837be-e9a7-4781-bb0d-8b7a50f63555-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9f837be-e9a7-4781-bb0d-8b7a50f63555-workload-certs\") pod \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\" (UID: \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\") "
Nov 03 09:17:03.245694 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:03.245231    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9f837be-e9a7-4781-bb0d-8b7a50f63555-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9f837be-e9a7-4781-bb0d-8b7a50f63555-istio-data\") pod \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\" (UID: \"e9f837be-e9a7-4781-bb0d-8b7a50f63555\") "
Nov 03 09:17:03.245694 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:03.245442    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9f837be-e9a7-4781-bb0d-8b7a50f63555-istiod-ca-cert" pod "e9f837be-e9a7-4781-bb0d-8b7a50f63555" (UID: "e9f837be-e9a7-4781-bb0d-8b7a50f63555"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:17:03.245968 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:03.245880    4473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9f837be-e9a7-4781-bb0d-8b7a50f63555-istiod-ca-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:17:03.249967 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:03.246514    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9f837be-e9a7-4781-bb0d-8b7a50f63555-workload-socket" pod "e9f837be-e9a7-4781-bb0d-8b7a50f63555" (UID: "e9f837be-e9a7-4781-bb0d-8b7a50f63555"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:17:03.249967 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:03.247858    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9f837be-e9a7-4781-bb0d-8b7a50f63555-workload-certs" pod "e9f837be-e9a7-4781-bb0d-8b7a50f63555" (UID: "e9f837be-e9a7-4781-bb0d-8b7a50f63555"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:17:03.254967 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:03.253122    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9f837be-e9a7-4781-bb0d-8b7a50f63555-credential-socket" pod "e9f837be-e9a7-4781-bb0d-8b7a50f63555" (UID: "e9f837be-e9a7-4781-bb0d-8b7a50f63555"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:17:03.258903 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:03.256804    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9f837be-e9a7-4781-bb0d-8b7a50f63555-istio-data" pod "e9f837be-e9a7-4781-bb0d-8b7a50f63555" (UID: "e9f837be-e9a7-4781-bb0d-8b7a50f63555"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:17:03.259923 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:03.259805    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9f837be-e9a7-4781-bb0d-8b7a50f63555-istio-token" pod "e9f837be-e9a7-4781-bb0d-8b7a50f63555" (UID: "e9f837be-e9a7-4781-bb0d-8b7a50f63555"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:17:03.261497 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:03.261460    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9f837be-e9a7-4781-bb0d-8b7a50f63555-istio-podinfo" pod "e9f837be-e9a7-4781-bb0d-8b7a50f63555" (UID: "e9f837be-e9a7-4781-bb0d-8b7a50f63555"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 09:17:03.262342 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:03.262312    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9f837be-e9a7-4781-bb0d-8b7a50f63555-kube-api-access-dht5v" pod "e9f837be-e9a7-4781-bb0d-8b7a50f63555" (UID: "e9f837be-e9a7-4781-bb0d-8b7a50f63555"). InnerVolumeSpecName "kube-api-access-dht5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:17:03.262963 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:03.262921    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9f837be-e9a7-4781-bb0d-8b7a50f63555-istio-envoy" pod "e9f837be-e9a7-4781-bb0d-8b7a50f63555" (UID: "e9f837be-e9a7-4781-bb0d-8b7a50f63555"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:17:03.347513 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:03.347482    4473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9f837be-e9a7-4781-bb0d-8b7a50f63555-istio-podinfo\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:17:03.347513 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:03.347512    4473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9f837be-e9a7-4781-bb0d-8b7a50f63555-workload-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:17:03.347735 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:03.347524    4473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9f837be-e9a7-4781-bb0d-8b7a50f63555-istio-data\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:17:03.347735 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:03.347538    4473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9f837be-e9a7-4781-bb0d-8b7a50f63555-credential-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:17:03.347735 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:03.347549    4473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9f837be-e9a7-4781-bb0d-8b7a50f63555-workload-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:17:03.347735 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:03.347561    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dht5v\" (UniqueName: \"kubernetes.io/projected/e9f837be-e9a7-4781-bb0d-8b7a50f63555-kube-api-access-dht5v\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:17:03.347735 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:03.347573    4473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9f837be-e9a7-4781-bb0d-8b7a50f63555-istio-token\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:17:03.347735 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:03.347585    4473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9f837be-e9a7-4781-bb0d-8b7a50f63555-istio-envoy\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:17:04.024242 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:04.024210    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"
Nov 03 09:17:04.024390 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:04.024237    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66" event={"ID":"e9f837be-e9a7-4781-bb0d-8b7a50f63555","Type":"ContainerDied","Data":"ec5758cf3e2a8831baa6bcd4e7f8344a774df7054e4293608a025144f1bc4c10"}
Nov 03 09:17:04.024390 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:04.024300    4473 scope.go:122] "RemoveContainer" containerID="b61b53e74603e54735baf74ef1b15bb2d57990d04f0af046aef41a16a84bd753"
Nov 03 09:17:04.195592 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:04.195514    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"]
Nov 03 09:17:04.246147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:04.246117    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-98d5dd-openshift-default-9b64c5595-ktf66"]
Nov 03 09:17:05.869649 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:05.869614    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9f837be-e9a7-4781-bb0d-8b7a50f63555" path="/var/lib/kubelet/pods/e9f837be-e9a7-4781-bb0d-8b7a50f63555/volumes"
Nov 03 09:17:06.651615 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:06.651585    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:17:09.807970 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:09.778052    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx" podUID="8e84a474-def6-41dc-8516-5e5ccf42ff46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:17:09.807970 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:09.778474    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5" podUID="2e42501e-d39d-4ae1-91c7-76c3c171538e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded"
Nov 03 09:17:09.808789 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:09.808664    4473 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5" podUID="2e42501e-d39d-4ae1-91c7-76c3c171538e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:17:09.851977 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:09.850177    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g" podUID="9e7f39b3-ce66-4b2d-abbc-8f078d527a2d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:17:09.920463 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:09.920415    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn" podUID="035b0461-2dfa-42f4-8ea7-2d1911cddca9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 09:17:09.955533 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:09.953449    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh" podUID="00bab831-c667-415d-b6b0-81fff4d57762" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:17:10.026305 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:10.026261    4473 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6" podUID="25c366b3-b919-462b-b69a-82ce9dbd7436" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:17:10.159976 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:10.159917    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn" podUID="05a8a676-3011-4597-9b71-b70c3387e2b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:17:10.248082 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:10.248036    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj" podUID="6afd6431-8c5e-4a46-b505-270efaaacd1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": context deadline exceeded"
Nov 03 09:17:10.428030 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:10.426163    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6" podUID="25c366b3-b919-462b-b69a-82ce9dbd7436" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:17:10.785021 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:10.784565    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx" podUID="8e84a474-def6-41dc-8516-5e5ccf42ff46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:17:10.793541 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:10.788819    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx"
Nov 03 09:17:10.827966 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:10.825385    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx"
Nov 03 09:17:16.941425 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:17:16.941381    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:17:24.737300 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:24.737267    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-0eecfd-76cbfb4455-4ztkx"]
Nov 03 09:17:27.353808 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:27.353776    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-048c3973-7q8d2"]
Nov 03 09:17:27.354318 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:27.354304    4473 cpu_manager.go:414] "containerMap: removing container" podUID="24ee886a-7918-4021-a818-0cdea41cf044" containerName="revoke-keys"
Nov 03 09:17:27.354370 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:27.354319    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24ee886a-7918-4021-a818-0cdea41cf044" containerName="revoke-keys"
Nov 03 09:17:27.354370 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:27.354347    4473 cpu_manager.go:414] "containerMap: removing container" podUID="e9f837be-e9a7-4781-bb0d-8b7a50f63555" containerName="istio-proxy"
Nov 03 09:17:27.354370 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:27.354352    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9f837be-e9a7-4781-bb0d-8b7a50f63555" containerName="istio-proxy"
Nov 03 09:17:27.354472 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:27.354427    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9f837be-e9a7-4781-bb0d-8b7a50f63555" containerName="istio-proxy"
Nov 03 09:17:27.354472 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:27.354441    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="24ee886a-7918-4021-a818-0cdea41cf044" containerName="revoke-keys"
Nov 03 09:17:27.357804 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:27.357786    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048c3973-7q8d2"
Nov 03 09:17:27.412864 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:27.412833    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-048c3973-7q8d2\" (UID: \"bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048c3973-7q8d2"
Nov 03 09:17:27.413062 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:27.412908    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgftd\" (UniqueName: \"kubernetes.io/projected/bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c-kube-api-access-wgftd\") pod \"maas-api-revoke-sub-048c3973-7q8d2\" (UID: \"bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048c3973-7q8d2"
Nov 03 09:17:27.413199 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:27.413175    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-048c3973-7q8d2\" (UID: \"bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048c3973-7q8d2"
Nov 03 09:17:27.416744 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:27.416722    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-048c3973-7q8d2"]
Nov 03 09:17:27.462298 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:27.462269    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:17:27.462298 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:27.462286    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:17:27.462298 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:27.462302    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:17:27.462298 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:27.462286    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:17:27.514325 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:27.514289    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-048c3973-7q8d2\" (UID: \"bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048c3973-7q8d2"
Nov 03 09:17:27.514512 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:27.514353    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgftd\" (UniqueName: \"kubernetes.io/projected/bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c-kube-api-access-wgftd\") pod \"maas-api-revoke-sub-048c3973-7q8d2\" (UID: \"bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048c3973-7q8d2"
Nov 03 09:17:27.514577 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:27.514538    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-048c3973-7q8d2\" (UID: \"bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048c3973-7q8d2"
Nov 03 09:17:27.515338 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:27.515311    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-048c3973-7q8d2\" (UID: \"bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048c3973-7q8d2"
Nov 03 09:17:27.516431 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:27.516404    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59f25a66-r8zcd"]
Nov 03 09:17:27.517045 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:27.517024    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-048c3973-7q8d2\" (UID: \"bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048c3973-7q8d2"
Nov 03 09:17:27.520330 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:27.520315    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f25a66-r8zcd"
Nov 03 09:17:27.571960 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:27.571915    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59f25a66-r8zcd"]
Nov 03 09:17:27.579147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:27.579120    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgftd\" (UniqueName: \"kubernetes.io/projected/bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c-kube-api-access-wgftd\") pod \"maas-api-revoke-sub-048c3973-7q8d2\" (UID: \"bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048c3973-7q8d2"
Nov 03 09:17:27.616096 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:27.616063    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2206c986-02c0-4e5b-8c37-13ce79cf063a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59f25a66-r8zcd\" (UID: \"2206c986-02c0-4e5b-8c37-13ce79cf063a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f25a66-r8zcd"
Nov 03 09:17:27.616309 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:27.616292    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdflr\" (UniqueName: \"kubernetes.io/projected/2206c986-02c0-4e5b-8c37-13ce79cf063a-kube-api-access-sdflr\") pod \"maas-api-revoke-sub-59f25a66-r8zcd\" (UID: \"2206c986-02c0-4e5b-8c37-13ce79cf063a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f25a66-r8zcd"
Nov 03 09:17:27.616382 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:27.616340    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2206c986-02c0-4e5b-8c37-13ce79cf063a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59f25a66-r8zcd\" (UID: \"2206c986-02c0-4e5b-8c37-13ce79cf063a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f25a66-r8zcd"
Nov 03 09:17:27.671421 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:27.671391    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048c3973-7q8d2"
Nov 03 09:17:27.717271 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:27.717238    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdflr\" (UniqueName: \"kubernetes.io/projected/2206c986-02c0-4e5b-8c37-13ce79cf063a-kube-api-access-sdflr\") pod \"maas-api-revoke-sub-59f25a66-r8zcd\" (UID: \"2206c986-02c0-4e5b-8c37-13ce79cf063a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f25a66-r8zcd"
Nov 03 09:17:27.717455 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:27.717284    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2206c986-02c0-4e5b-8c37-13ce79cf063a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59f25a66-r8zcd\" (UID: \"2206c986-02c0-4e5b-8c37-13ce79cf063a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f25a66-r8zcd"
Nov 03 09:17:27.717455 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:27.717335    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2206c986-02c0-4e5b-8c37-13ce79cf063a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59f25a66-r8zcd\" (UID: \"2206c986-02c0-4e5b-8c37-13ce79cf063a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f25a66-r8zcd"
Nov 03 09:17:27.718027 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:27.718008    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2206c986-02c0-4e5b-8c37-13ce79cf063a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59f25a66-r8zcd\" (UID: \"2206c986-02c0-4e5b-8c37-13ce79cf063a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f25a66-r8zcd"
Nov 03 09:17:27.720038 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:27.720017    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2206c986-02c0-4e5b-8c37-13ce79cf063a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59f25a66-r8zcd\" (UID: \"2206c986-02c0-4e5b-8c37-13ce79cf063a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f25a66-r8zcd"
Nov 03 09:17:27.777702 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:27.777669    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdflr\" (UniqueName: \"kubernetes.io/projected/2206c986-02c0-4e5b-8c37-13ce79cf063a-kube-api-access-sdflr\") pod \"maas-api-revoke-sub-59f25a66-r8zcd\" (UID: \"2206c986-02c0-4e5b-8c37-13ce79cf063a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f25a66-r8zcd"
Nov 03 09:17:27.842181 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:27.842150    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f25a66-r8zcd"
Nov 03 09:17:28.048442 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:28.048416    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-048c3973-7q8d2"]
Nov 03 09:17:28.048979 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:17:28.048927    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac2b2a4_3bbc_4729_81c0_1c68fdabcf6c.slice/crio-2fa8f63de4e154eb073be75b947c4dbd8496223f9f9f19f5a0d3d59931eee8ca WatchSource:0}: Error finding container 2fa8f63de4e154eb073be75b947c4dbd8496223f9f9f19f5a0d3d59931eee8ca: Status 404 returned error can't find the container with id 2fa8f63de4e154eb073be75b947c4dbd8496223f9f9f19f5a0d3d59931eee8ca
Nov 03 09:17:28.218530 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:28.218501    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59f25a66-r8zcd"]
Nov 03 09:17:28.219665 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:17:28.219635    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2206c986_02c0_4e5b_8c37_13ce79cf063a.slice/crio-1088137316d4cfe76f3b9bd591f5f40f508179c17feafeb5b7b96302b179e93a WatchSource:0}: Error finding container 1088137316d4cfe76f3b9bd591f5f40f508179c17feafeb5b7b96302b179e93a: Status 404 returned error can't find the container with id 1088137316d4cfe76f3b9bd591f5f40f508179c17feafeb5b7b96302b179e93a
Nov 03 09:17:28.251018 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:28.250980    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048c3973-7q8d2" event={"ID":"bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c","Type":"ContainerStarted","Data":"2fa8f63de4e154eb073be75b947c4dbd8496223f9f9f19f5a0d3d59931eee8ca"}
Nov 03 09:17:28.252316 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:28.252291    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f25a66-r8zcd" event={"ID":"2206c986-02c0-4e5b-8c37-13ce79cf063a","Type":"ContainerStarted","Data":"1088137316d4cfe76f3b9bd591f5f40f508179c17feafeb5b7b96302b179e93a"}
Nov 03 09:17:29.261255 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:29.261159    4473 generic.go:356] "Generic (PLEG): container finished" podID="bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c" containerID="107571e02772fc15774f22d4d34b91c90c1e92acb6a957f39057012877932977" exitCode=0
Nov 03 09:17:29.261651 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:29.261246    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048c3973-7q8d2" event={"ID":"bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c","Type":"ContainerDied","Data":"107571e02772fc15774f22d4d34b91c90c1e92acb6a957f39057012877932977"}
Nov 03 09:17:29.262875 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:29.262849    4473 generic.go:356] "Generic (PLEG): container finished" podID="2206c986-02c0-4e5b-8c37-13ce79cf063a" containerID="68cde13d9a5b2d18d806b006161874cb9fae146633eb0a2b1368a333bba331a4" exitCode=0
Nov 03 09:17:29.263007 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:29.262919    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f25a66-r8zcd" event={"ID":"2206c986-02c0-4e5b-8c37-13ce79cf063a","Type":"ContainerDied","Data":"68cde13d9a5b2d18d806b006161874cb9fae146633eb0a2b1368a333bba331a4"}
Nov 03 09:17:29.737961 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:29.737910    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0eecfd-76cbfb4455-4ztkx" podUID="6cb96f19-e812-48af-9995-3fdc60b20e15" containerName="payload-pre-processing" containerID="cri-o://5a45530a76d4c05527d7761c70e208faed5f9eb44898a9ebe2cbb21fba29361d" gracePeriod=25
Nov 03 09:17:30.007322 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:30.007298    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0eecfd-76cbfb4455-4ztkx"
Nov 03 09:17:30.045261 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:30.045227    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cb96f19-e812-48af-9995-3fdc60b20e15-kube-api-access-bjvrv\" (UniqueName: \"kubernetes.io/projected/6cb96f19-e812-48af-9995-3fdc60b20e15-kube-api-access-bjvrv\") pod \"6cb96f19-e812-48af-9995-3fdc60b20e15\" (UID: \"6cb96f19-e812-48af-9995-3fdc60b20e15\") "
Nov 03 09:17:30.045437 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:30.045423    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cb96f19-e812-48af-9995-3fdc60b20e15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb96f19-e812-48af-9995-3fdc60b20e15-plugins-config-volume\") pod \"6cb96f19-e812-48af-9995-3fdc60b20e15\" (UID: \"6cb96f19-e812-48af-9995-3fdc60b20e15\") "
Nov 03 09:17:30.045803 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:30.045779    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cb96f19-e812-48af-9995-3fdc60b20e15-plugins-config-volume" pod "6cb96f19-e812-48af-9995-3fdc60b20e15" (UID: "6cb96f19-e812-48af-9995-3fdc60b20e15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:17:30.047537 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:30.047511    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cb96f19-e812-48af-9995-3fdc60b20e15-kube-api-access-bjvrv" pod "6cb96f19-e812-48af-9995-3fdc60b20e15" (UID: "6cb96f19-e812-48af-9995-3fdc60b20e15"). InnerVolumeSpecName "kube-api-access-bjvrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:17:30.147007 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:30.146975    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb96f19-e812-48af-9995-3fdc60b20e15-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:17:30.147007 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:30.147005    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjvrv\" (UniqueName: \"kubernetes.io/projected/6cb96f19-e812-48af-9995-3fdc60b20e15-kube-api-access-bjvrv\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:17:30.270604 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:30.270519    4473 generic.go:356] "Generic (PLEG): container finished" podID="6cb96f19-e812-48af-9995-3fdc60b20e15" containerID="5a45530a76d4c05527d7761c70e208faed5f9eb44898a9ebe2cbb21fba29361d" exitCode=0
Nov 03 09:17:30.270604 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:30.270580    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0eecfd-76cbfb4455-4ztkx"
Nov 03 09:17:30.271155 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:30.270602    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0eecfd-76cbfb4455-4ztkx" event={"ID":"6cb96f19-e812-48af-9995-3fdc60b20e15","Type":"ContainerDied","Data":"5a45530a76d4c05527d7761c70e208faed5f9eb44898a9ebe2cbb21fba29361d"}
Nov 03 09:17:30.271155 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:30.270675    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0eecfd-76cbfb4455-4ztkx" event={"ID":"6cb96f19-e812-48af-9995-3fdc60b20e15","Type":"ContainerDied","Data":"04f13ead6ec475f5bca6973ea500da34802048a4dd7af0100fd56d64f9c6d545"}
Nov 03 09:17:30.271155 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:30.270696    4473 scope.go:122] "RemoveContainer" containerID="5a45530a76d4c05527d7761c70e208faed5f9eb44898a9ebe2cbb21fba29361d"
Nov 03 09:17:30.284381 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:30.284358    4473 scope.go:122] "RemoveContainer" containerID="5a45530a76d4c05527d7761c70e208faed5f9eb44898a9ebe2cbb21fba29361d"
Nov 03 09:17:30.284720 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:17:30.284698    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a45530a76d4c05527d7761c70e208faed5f9eb44898a9ebe2cbb21fba29361d\": container with ID starting with 5a45530a76d4c05527d7761c70e208faed5f9eb44898a9ebe2cbb21fba29361d not found: ID does not exist" containerID="5a45530a76d4c05527d7761c70e208faed5f9eb44898a9ebe2cbb21fba29361d"
Nov 03 09:17:30.284820 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:30.284727    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a45530a76d4c05527d7761c70e208faed5f9eb44898a9ebe2cbb21fba29361d"} err="failed to get container status \"5a45530a76d4c05527d7761c70e208faed5f9eb44898a9ebe2cbb21fba29361d\": rpc error: code = NotFound desc = could not find container \"5a45530a76d4c05527d7761c70e208faed5f9eb44898a9ebe2cbb21fba29361d\": container with ID starting with 5a45530a76d4c05527d7761c70e208faed5f9eb44898a9ebe2cbb21fba29361d not found: ID does not exist"
Nov 03 09:17:30.420770 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:30.420743    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048c3973-7q8d2"
Nov 03 09:17:30.440198 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:30.440177    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f25a66-r8zcd"
Nov 03 09:17:30.441177 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:30.441157    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-0eecfd-76cbfb4455-4ztkx"]
Nov 03 09:17:30.448725 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:30.448704    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-0eecfd-76cbfb4455-4ztkx"]
Nov 03 09:17:30.552022 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:30.551929    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2206c986-02c0-4e5b-8c37-13ce79cf063a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2206c986-02c0-4e5b-8c37-13ce79cf063a-maas-api-xks-ca\") pod \"2206c986-02c0-4e5b-8c37-13ce79cf063a\" (UID: \"2206c986-02c0-4e5b-8c37-13ce79cf063a\") "
Nov 03 09:17:30.552022 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:30.552007    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c-kube-api-access-wgftd\" (UniqueName: \"kubernetes.io/projected/bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c-kube-api-access-wgftd\") pod \"bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c\" (UID: \"bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c\") "
Nov 03 09:17:30.552254 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:30.552054    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c-maas-api-openshift-ca\") pod \"bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c\" (UID: \"bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c\") "
Nov 03 09:17:30.552254 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:30.552104    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2206c986-02c0-4e5b-8c37-13ce79cf063a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2206c986-02c0-4e5b-8c37-13ce79cf063a-maas-api-openshift-ca\") pod \"2206c986-02c0-4e5b-8c37-13ce79cf063a\" (UID: \"2206c986-02c0-4e5b-8c37-13ce79cf063a\") "
Nov 03 09:17:30.552254 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:30.552136    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2206c986-02c0-4e5b-8c37-13ce79cf063a-kube-api-access-sdflr\" (UniqueName: \"kubernetes.io/projected/2206c986-02c0-4e5b-8c37-13ce79cf063a-kube-api-access-sdflr\") pod \"2206c986-02c0-4e5b-8c37-13ce79cf063a\" (UID: \"2206c986-02c0-4e5b-8c37-13ce79cf063a\") "
Nov 03 09:17:30.552254 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:30.552159    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c-maas-api-xks-ca\") pod \"bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c\" (UID: \"bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c\") "
Nov 03 09:17:30.552506 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:30.552480    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c-maas-api-openshift-ca" pod "bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c" (UID: "bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:17:30.552570 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:30.552511    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2206c986-02c0-4e5b-8c37-13ce79cf063a-maas-api-openshift-ca" pod "2206c986-02c0-4e5b-8c37-13ce79cf063a" (UID: "2206c986-02c0-4e5b-8c37-13ce79cf063a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:17:30.552762 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:30.552711    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2206c986-02c0-4e5b-8c37-13ce79cf063a-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:17:30.552762 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:30.552732    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:17:30.554342 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:30.554307    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2206c986-02c0-4e5b-8c37-13ce79cf063a-maas-api-xks-ca" pod "2206c986-02c0-4e5b-8c37-13ce79cf063a" (UID: "2206c986-02c0-4e5b-8c37-13ce79cf063a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:17:30.554441 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:30.554387    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c-kube-api-access-wgftd" pod "bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c" (UID: "bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c"). InnerVolumeSpecName "kube-api-access-wgftd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:17:30.554441 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:30.554421    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2206c986-02c0-4e5b-8c37-13ce79cf063a-kube-api-access-sdflr" pod "2206c986-02c0-4e5b-8c37-13ce79cf063a" (UID: "2206c986-02c0-4e5b-8c37-13ce79cf063a"). InnerVolumeSpecName "kube-api-access-sdflr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:17:30.554561 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:30.554544    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c-maas-api-xks-ca" pod "bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c" (UID: "bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:17:30.653713 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:30.653682    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2206c986-02c0-4e5b-8c37-13ce79cf063a-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:17:30.653713 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:30.653708    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgftd\" (UniqueName: \"kubernetes.io/projected/bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c-kube-api-access-wgftd\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:17:30.653713 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:30.653719    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdflr\" (UniqueName: \"kubernetes.io/projected/2206c986-02c0-4e5b-8c37-13ce79cf063a-kube-api-access-sdflr\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:17:30.653934 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:30.653729    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:17:31.279649 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:31.279616    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048c3973-7q8d2"
Nov 03 09:17:31.280116 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:31.279650    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048c3973-7q8d2" event={"ID":"bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c","Type":"ContainerDied","Data":"2fa8f63de4e154eb073be75b947c4dbd8496223f9f9f19f5a0d3d59931eee8ca"}
Nov 03 09:17:31.280116 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:31.279707    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fa8f63de4e154eb073be75b947c4dbd8496223f9f9f19f5a0d3d59931eee8ca"
Nov 03 09:17:31.281060 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:31.281045    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f25a66-r8zcd"
Nov 03 09:17:31.281060 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:31.281047    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59f25a66-r8zcd" event={"ID":"2206c986-02c0-4e5b-8c37-13ce79cf063a","Type":"ContainerDied","Data":"1088137316d4cfe76f3b9bd591f5f40f508179c17feafeb5b7b96302b179e93a"}
Nov 03 09:17:31.281178 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:31.281089    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1088137316d4cfe76f3b9bd591f5f40f508179c17feafeb5b7b96302b179e93a"
Nov 03 09:17:31.870290 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:31.870255    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cb96f19-e812-48af-9995-3fdc60b20e15" path="/var/lib/kubelet/pods/6cb96f19-e812-48af-9995-3fdc60b20e15/volumes"
Nov 03 09:17:34.447697 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:34.447657    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e762eaf6-hnk4r"]
Nov 03 09:17:34.448978 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:34.448935    4473 cpu_manager.go:414] "containerMap: removing container" podUID="6cb96f19-e812-48af-9995-3fdc60b20e15" containerName="payload-pre-processing"
Nov 03 09:17:34.453001 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:34.452976    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cb96f19-e812-48af-9995-3fdc60b20e15" containerName="payload-pre-processing"
Nov 03 09:17:34.453184 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:34.453170    4473 cpu_manager.go:414] "containerMap: removing container" podUID="2206c986-02c0-4e5b-8c37-13ce79cf063a" containerName="revoke-keys"
Nov 03 09:17:34.453282 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:34.453271    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2206c986-02c0-4e5b-8c37-13ce79cf063a" containerName="revoke-keys"
Nov 03 09:17:34.453377 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:34.453368    4473 cpu_manager.go:414] "containerMap: removing container" podUID="bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c" containerName="revoke-keys"
Nov 03 09:17:34.453448 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:34.453439    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c" containerName="revoke-keys"
Nov 03 09:17:34.461199 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:34.461171    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cb96f19-e812-48af-9995-3fdc60b20e15" containerName="payload-pre-processing"
Nov 03 09:17:34.461357 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:34.461345    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="2206c986-02c0-4e5b-8c37-13ce79cf063a" containerName="revoke-keys"
Nov 03 09:17:34.461438 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:34.461430    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c" containerName="revoke-keys"
Nov 03 09:17:34.488969 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:34.485279    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e762eaf6-hnk4r"
Nov 03 09:17:34.524196 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:34.524157    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e762eaf6-hnk4r"]
Nov 03 09:17:34.558014 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:34.557988    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:17:34.587081 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:34.568785    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:17:34.587424 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:34.568880    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:17:34.596173 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:34.596145    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:17:34.675706 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:34.675585    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b3ced30-0ba0-45b9-b1fc-49b730f797c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e762eaf6-hnk4r\" (UID: \"1b3ced30-0ba0-45b9-b1fc-49b730f797c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e762eaf6-hnk4r"
Nov 03 09:17:34.676084 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:34.676066    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b3ced30-0ba0-45b9-b1fc-49b730f797c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e762eaf6-hnk4r\" (UID: \"1b3ced30-0ba0-45b9-b1fc-49b730f797c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e762eaf6-hnk4r"
Nov 03 09:17:34.676562 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:34.676545    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbc9h\" (UniqueName: \"kubernetes.io/projected/1b3ced30-0ba0-45b9-b1fc-49b730f797c1-kube-api-access-mbc9h\") pod \"maas-api-revoke-sub-e762eaf6-hnk4r\" (UID: \"1b3ced30-0ba0-45b9-b1fc-49b730f797c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e762eaf6-hnk4r"
Nov 03 09:17:34.787602 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:34.787489    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b3ced30-0ba0-45b9-b1fc-49b730f797c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e762eaf6-hnk4r\" (UID: \"1b3ced30-0ba0-45b9-b1fc-49b730f797c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e762eaf6-hnk4r"
Nov 03 09:17:34.817472 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:34.817425    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b3ced30-0ba0-45b9-b1fc-49b730f797c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e762eaf6-hnk4r\" (UID: \"1b3ced30-0ba0-45b9-b1fc-49b730f797c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e762eaf6-hnk4r"
Nov 03 09:17:34.817872 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:34.817848    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbc9h\" (UniqueName: \"kubernetes.io/projected/1b3ced30-0ba0-45b9-b1fc-49b730f797c1-kube-api-access-mbc9h\") pod \"maas-api-revoke-sub-e762eaf6-hnk4r\" (UID: \"1b3ced30-0ba0-45b9-b1fc-49b730f797c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e762eaf6-hnk4r"
Nov 03 09:17:34.957455 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:34.957416    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b3ced30-0ba0-45b9-b1fc-49b730f797c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e762eaf6-hnk4r\" (UID: \"1b3ced30-0ba0-45b9-b1fc-49b730f797c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e762eaf6-hnk4r"
Nov 03 09:17:35.001408 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:34.964597    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbc9h\" (UniqueName: \"kubernetes.io/projected/1b3ced30-0ba0-45b9-b1fc-49b730f797c1-kube-api-access-mbc9h\") pod \"maas-api-revoke-sub-e762eaf6-hnk4r\" (UID: \"1b3ced30-0ba0-45b9-b1fc-49b730f797c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e762eaf6-hnk4r"
Nov 03 09:17:35.001408 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:34.986414    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b3ced30-0ba0-45b9-b1fc-49b730f797c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e762eaf6-hnk4r\" (UID: \"1b3ced30-0ba0-45b9-b1fc-49b730f797c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e762eaf6-hnk4r"
Nov 03 09:17:35.137976 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:35.135969    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e762eaf6-hnk4r"
Nov 03 09:17:35.715973 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:35.715037    4473 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn" podUID="035b0461-2dfa-42f4-8ea7-2d1911cddca9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": context deadline exceeded"
Nov 03 09:17:35.789022 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:35.769097    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5" podUID="2e42501e-d39d-4ae1-91c7-76c3c171538e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:17:36.232220 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:36.232176    4473 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm" podUID="60b1d7ef-16c6-4b6a-973b-4a251593eac7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:17:36.232459 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:35.848553    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g" podUID="9e7f39b3-ce66-4b2d-abbc-8f078d527a2d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:17:36.235292 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:35.997431    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn" podUID="035b0461-2dfa-42f4-8ea7-2d1911cddca9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Nov 03 09:17:36.235458 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:36.009621    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd" podUID="cf4cf2fc-612a-4f5c-a596-543036dde7e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Nov 03 09:17:36.313140 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:36.313005    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj" podUID="6afd6431-8c5e-4a46-b505-270efaaacd1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.87:8000/ready\": context deadline exceeded"
Nov 03 09:17:36.338215 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:36.176374    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn" podUID="05a8a676-3011-4597-9b71-b70c3387e2b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:17:36.364388 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:35.846148    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx" podUID="8e84a474-def6-41dc-8516-5e5ccf42ff46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Nov 03 09:17:36.364388 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:36.127487    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh" podUID="00bab831-c667-415d-b6b0-81fff4d57762" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 03 09:17:36.479971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:36.456829    4473 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd" podUID="cf4cf2fc-612a-4f5c-a596-543036dde7e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:17:36.479971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:36.457074    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6" podUID="25c366b3-b919-462b-b69a-82ce9dbd7436" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:17:36.516022 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:36.508321    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e762eaf6-hnk4r"]
Nov 03 09:17:36.878782 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:36.878724    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e762eaf6-hnk4r" event={"ID":"1b3ced30-0ba0-45b9-b1fc-49b730f797c1","Type":"ContainerStarted","Data":"6778917ad0c215e1d4452c67960998332bd9d2bb68d1c9a2973301a4e0395f62"}
Nov 03 09:17:37.218856 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:37.218814    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5" podUID="2e42501e-d39d-4ae1-91c7-76c3c171538e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:17:37.224811 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:37.224786    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5"
Nov 03 09:17:37.244843 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:37.244808    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd" podUID="cf4cf2fc-612a-4f5c-a596-543036dde7e3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:17:37.246551 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:37.246530    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd"
Nov 03 09:17:37.395023 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:37.394980    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd"
Nov 03 09:17:37.409214 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:37.409188    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5"
Nov 03 09:17:37.886312 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:37.886269    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e762eaf6-hnk4r" event={"ID":"1b3ced30-0ba0-45b9-b1fc-49b730f797c1","Type":"ContainerStarted","Data":"6e89a8c8de59433d0d50d85f9ee0acbc0035ff7281de3a8a612b2011ec7b2aef"}
Nov 03 09:17:38.002556 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:38.002506    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e762eaf6-hnk4r" podStartSLOduration=4.002493581 podStartE2EDuration="4.002493581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:17:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:17:38.000895045 +0000 UTC m=+2282.269958480" watchObservedRunningTime="2026-11-03 09:17:38.002493581 +0000 UTC m=+2282.271557015"
Nov 03 09:17:38.313161 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:38.313083    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7188c7f1-5x2jb"]
Nov 03 09:17:38.317371 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:38.317348    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7188c7f1-5x2jb"
Nov 03 09:17:38.374721 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:38.374696    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7188c7f1-5x2jb"]
Nov 03 09:17:38.426527 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:38.426497    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5ptr\" (UniqueName: \"kubernetes.io/projected/a64fae7c-510c-41a3-8a75-95203daac0bf-kube-api-access-f5ptr\") pod \"maas-api-revoke-sub-7188c7f1-5x2jb\" (UID: \"a64fae7c-510c-41a3-8a75-95203daac0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7188c7f1-5x2jb"
Nov 03 09:17:38.426527 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:38.426533    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a64fae7c-510c-41a3-8a75-95203daac0bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7188c7f1-5x2jb\" (UID: \"a64fae7c-510c-41a3-8a75-95203daac0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7188c7f1-5x2jb"
Nov 03 09:17:38.426753 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:38.426626    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a64fae7c-510c-41a3-8a75-95203daac0bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7188c7f1-5x2jb\" (UID: \"a64fae7c-510c-41a3-8a75-95203daac0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7188c7f1-5x2jb"
Nov 03 09:17:38.527549 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:38.527515    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a64fae7c-510c-41a3-8a75-95203daac0bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7188c7f1-5x2jb\" (UID: \"a64fae7c-510c-41a3-8a75-95203daac0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7188c7f1-5x2jb"
Nov 03 09:17:38.527730 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:38.527711    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5ptr\" (UniqueName: \"kubernetes.io/projected/a64fae7c-510c-41a3-8a75-95203daac0bf-kube-api-access-f5ptr\") pod \"maas-api-revoke-sub-7188c7f1-5x2jb\" (UID: \"a64fae7c-510c-41a3-8a75-95203daac0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7188c7f1-5x2jb"
Nov 03 09:17:38.527804 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:38.527745    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a64fae7c-510c-41a3-8a75-95203daac0bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7188c7f1-5x2jb\" (UID: \"a64fae7c-510c-41a3-8a75-95203daac0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7188c7f1-5x2jb"
Nov 03 09:17:38.528370 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:38.528345    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a64fae7c-510c-41a3-8a75-95203daac0bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7188c7f1-5x2jb\" (UID: \"a64fae7c-510c-41a3-8a75-95203daac0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7188c7f1-5x2jb"
Nov 03 09:17:38.530023 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:38.529999    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a64fae7c-510c-41a3-8a75-95203daac0bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7188c7f1-5x2jb\" (UID: \"a64fae7c-510c-41a3-8a75-95203daac0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7188c7f1-5x2jb"
Nov 03 09:17:38.586496 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:38.586440    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5ptr\" (UniqueName: \"kubernetes.io/projected/a64fae7c-510c-41a3-8a75-95203daac0bf-kube-api-access-f5ptr\") pod \"maas-api-revoke-sub-7188c7f1-5x2jb\" (UID: \"a64fae7c-510c-41a3-8a75-95203daac0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7188c7f1-5x2jb"
Nov 03 09:17:38.630544 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:38.630504    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7188c7f1-5x2jb"
Nov 03 09:17:38.895212 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:38.895183    4473 generic.go:356] "Generic (PLEG): container finished" podID="1b3ced30-0ba0-45b9-b1fc-49b730f797c1" containerID="6e89a8c8de59433d0d50d85f9ee0acbc0035ff7281de3a8a612b2011ec7b2aef" exitCode=0
Nov 03 09:17:38.895569 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:38.895268    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e762eaf6-hnk4r" event={"ID":"1b3ced30-0ba0-45b9-b1fc-49b730f797c1","Type":"ContainerDied","Data":"6e89a8c8de59433d0d50d85f9ee0acbc0035ff7281de3a8a612b2011ec7b2aef"}
Nov 03 09:17:39.021804 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:39.021766    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7188c7f1-5x2jb"]
Nov 03 09:17:39.021979 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:17:39.021850    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda64fae7c_510c_41a3_8a75_95203daac0bf.slice/crio-15ee67fb027a6e6aadb3dc460dabfeb41dc75899856abe4fdb632d3e4bf10a21 WatchSource:0}: Error finding container 15ee67fb027a6e6aadb3dc460dabfeb41dc75899856abe4fdb632d3e4bf10a21: Status 404 returned error can't find the container with id 15ee67fb027a6e6aadb3dc460dabfeb41dc75899856abe4fdb632d3e4bf10a21
Nov 03 09:17:39.903071 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:39.903033    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7188c7f1-5x2jb" event={"ID":"a64fae7c-510c-41a3-8a75-95203daac0bf","Type":"ContainerStarted","Data":"ddf8ae3056e9f37019bfa451a7f7d5a929e145ef4e7113f94ec1f372f7b44b54"}
Nov 03 09:17:39.903526 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:39.903101    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7188c7f1-5x2jb" event={"ID":"a64fae7c-510c-41a3-8a75-95203daac0bf","Type":"ContainerStarted","Data":"15ee67fb027a6e6aadb3dc460dabfeb41dc75899856abe4fdb632d3e4bf10a21"}
Nov 03 09:17:40.021777 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:40.021719    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7188c7f1-5x2jb" podStartSLOduration=2.021701387 podStartE2EDuration="2.021701387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:17:40.019307239 +0000 UTC m=+2284.288370674" watchObservedRunningTime="2026-11-03 09:17:40.021701387 +0000 UTC m=+2284.290764822"
Nov 03 09:17:40.076866 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:40.076844    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e762eaf6-hnk4r"
Nov 03 09:17:40.148705 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:40.148668    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b3ced30-0ba0-45b9-b1fc-49b730f797c1-kube-api-access-mbc9h\" (UniqueName: \"kubernetes.io/projected/1b3ced30-0ba0-45b9-b1fc-49b730f797c1-kube-api-access-mbc9h\") pod \"1b3ced30-0ba0-45b9-b1fc-49b730f797c1\" (UID: \"1b3ced30-0ba0-45b9-b1fc-49b730f797c1\") "
Nov 03 09:17:40.148705 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:40.148715    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b3ced30-0ba0-45b9-b1fc-49b730f797c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b3ced30-0ba0-45b9-b1fc-49b730f797c1-maas-api-openshift-ca\") pod \"1b3ced30-0ba0-45b9-b1fc-49b730f797c1\" (UID: \"1b3ced30-0ba0-45b9-b1fc-49b730f797c1\") "
Nov 03 09:17:40.148919 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:40.148779    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b3ced30-0ba0-45b9-b1fc-49b730f797c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b3ced30-0ba0-45b9-b1fc-49b730f797c1-maas-api-xks-ca\") pod \"1b3ced30-0ba0-45b9-b1fc-49b730f797c1\" (UID: \"1b3ced30-0ba0-45b9-b1fc-49b730f797c1\") "
Nov 03 09:17:40.149201 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:40.149165    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b3ced30-0ba0-45b9-b1fc-49b730f797c1-maas-api-openshift-ca" pod "1b3ced30-0ba0-45b9-b1fc-49b730f797c1" (UID: "1b3ced30-0ba0-45b9-b1fc-49b730f797c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:17:40.150998 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:40.150973    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b3ced30-0ba0-45b9-b1fc-49b730f797c1-kube-api-access-mbc9h" pod "1b3ced30-0ba0-45b9-b1fc-49b730f797c1" (UID: "1b3ced30-0ba0-45b9-b1fc-49b730f797c1"). InnerVolumeSpecName "kube-api-access-mbc9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:17:40.151108 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:40.151034    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b3ced30-0ba0-45b9-b1fc-49b730f797c1-maas-api-xks-ca" pod "1b3ced30-0ba0-45b9-b1fc-49b730f797c1" (UID: "1b3ced30-0ba0-45b9-b1fc-49b730f797c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:17:40.250711 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:40.250676    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbc9h\" (UniqueName: \"kubernetes.io/projected/1b3ced30-0ba0-45b9-b1fc-49b730f797c1-kube-api-access-mbc9h\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:17:40.250711 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:40.250711    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b3ced30-0ba0-45b9-b1fc-49b730f797c1-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:17:40.250884 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:40.250729    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b3ced30-0ba0-45b9-b1fc-49b730f797c1-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:17:40.911883 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:40.911854    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e762eaf6-hnk4r"
Nov 03 09:17:40.912326 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:40.911894    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e762eaf6-hnk4r" event={"ID":"1b3ced30-0ba0-45b9-b1fc-49b730f797c1","Type":"ContainerDied","Data":"6778917ad0c215e1d4452c67960998332bd9d2bb68d1c9a2973301a4e0395f62"}
Nov 03 09:17:40.912326 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:40.911959    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6778917ad0c215e1d4452c67960998332bd9d2bb68d1c9a2973301a4e0395f62"
Nov 03 09:17:40.913905 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:40.913878    4473 generic.go:356] "Generic (PLEG): container finished" podID="a64fae7c-510c-41a3-8a75-95203daac0bf" containerID="ddf8ae3056e9f37019bfa451a7f7d5a929e145ef4e7113f94ec1f372f7b44b54" exitCode=0
Nov 03 09:17:40.914058 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:40.913936    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7188c7f1-5x2jb" event={"ID":"a64fae7c-510c-41a3-8a75-95203daac0bf","Type":"ContainerDied","Data":"ddf8ae3056e9f37019bfa451a7f7d5a929e145ef4e7113f94ec1f372f7b44b54"}
Nov 03 09:17:41.146891 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:41.146853    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00db84b1-hxwqt"]
Nov 03 09:17:41.147435 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:41.147420    4473 cpu_manager.go:414] "containerMap: removing container" podUID="1b3ced30-0ba0-45b9-b1fc-49b730f797c1" containerName="revoke-keys"
Nov 03 09:17:41.147486 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:41.147436    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b3ced30-0ba0-45b9-b1fc-49b730f797c1" containerName="revoke-keys"
Nov 03 09:17:41.147541 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:41.147532    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b3ced30-0ba0-45b9-b1fc-49b730f797c1" containerName="revoke-keys"
Nov 03 09:17:41.150953 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:41.150927    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00db84b1-hxwqt"
Nov 03 09:17:41.209193 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:41.209124    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00db84b1-hxwqt"]
Nov 03 09:17:41.260134 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:41.260107    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfb9ba89-9bc5-4024-b535-deeb4421b4e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00db84b1-hxwqt\" (UID: \"cfb9ba89-9bc5-4024-b535-deeb4421b4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00db84b1-hxwqt"
Nov 03 09:17:41.260287 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:41.260160    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzr99\" (UniqueName: \"kubernetes.io/projected/cfb9ba89-9bc5-4024-b535-deeb4421b4e2-kube-api-access-kzr99\") pod \"maas-api-revoke-sub-00db84b1-hxwqt\" (UID: \"cfb9ba89-9bc5-4024-b535-deeb4421b4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00db84b1-hxwqt"
Nov 03 09:17:41.260287 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:41.260261    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfb9ba89-9bc5-4024-b535-deeb4421b4e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00db84b1-hxwqt\" (UID: \"cfb9ba89-9bc5-4024-b535-deeb4421b4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00db84b1-hxwqt"
Nov 03 09:17:41.361423 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:41.361387    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfb9ba89-9bc5-4024-b535-deeb4421b4e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00db84b1-hxwqt\" (UID: \"cfb9ba89-9bc5-4024-b535-deeb4421b4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00db84b1-hxwqt"
Nov 03 09:17:41.361598 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:41.361454    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzr99\" (UniqueName: \"kubernetes.io/projected/cfb9ba89-9bc5-4024-b535-deeb4421b4e2-kube-api-access-kzr99\") pod \"maas-api-revoke-sub-00db84b1-hxwqt\" (UID: \"cfb9ba89-9bc5-4024-b535-deeb4421b4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00db84b1-hxwqt"
Nov 03 09:17:41.361598 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:41.361486    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfb9ba89-9bc5-4024-b535-deeb4421b4e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00db84b1-hxwqt\" (UID: \"cfb9ba89-9bc5-4024-b535-deeb4421b4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00db84b1-hxwqt"
Nov 03 09:17:41.362170 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:41.362145    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfb9ba89-9bc5-4024-b535-deeb4421b4e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00db84b1-hxwqt\" (UID: \"cfb9ba89-9bc5-4024-b535-deeb4421b4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00db84b1-hxwqt"
Nov 03 09:17:41.363882 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:41.363864    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfb9ba89-9bc5-4024-b535-deeb4421b4e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00db84b1-hxwqt\" (UID: \"cfb9ba89-9bc5-4024-b535-deeb4421b4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00db84b1-hxwqt"
Nov 03 09:17:41.421133 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:41.421098    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzr99\" (UniqueName: \"kubernetes.io/projected/cfb9ba89-9bc5-4024-b535-deeb4421b4e2-kube-api-access-kzr99\") pod \"maas-api-revoke-sub-00db84b1-hxwqt\" (UID: \"cfb9ba89-9bc5-4024-b535-deeb4421b4e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00db84b1-hxwqt"
Nov 03 09:17:41.464546 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:41.464481    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00db84b1-hxwqt"
Nov 03 09:17:41.843853 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:41.843824    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00db84b1-hxwqt"]
Nov 03 09:17:41.844484 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:17:41.844452    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfb9ba89_9bc5_4024_b535_deeb4421b4e2.slice/crio-03137da55e601d4160245ef51c22858f4f888152cf3c3506de5a53fbb82db34d WatchSource:0}: Error finding container 03137da55e601d4160245ef51c22858f4f888152cf3c3506de5a53fbb82db34d: Status 404 returned error can't find the container with id 03137da55e601d4160245ef51c22858f4f888152cf3c3506de5a53fbb82db34d
Nov 03 09:17:41.922864 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:41.922834    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00db84b1-hxwqt" event={"ID":"cfb9ba89-9bc5-4024-b535-deeb4421b4e2","Type":"ContainerStarted","Data":"03137da55e601d4160245ef51c22858f4f888152cf3c3506de5a53fbb82db34d"}
Nov 03 09:17:42.070312 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:42.070282    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7188c7f1-5x2jb"
Nov 03 09:17:42.188228 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:42.187083    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a64fae7c-510c-41a3-8a75-95203daac0bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a64fae7c-510c-41a3-8a75-95203daac0bf-maas-api-xks-ca\") pod \"a64fae7c-510c-41a3-8a75-95203daac0bf\" (UID: \"a64fae7c-510c-41a3-8a75-95203daac0bf\") "
Nov 03 09:17:42.188228 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:42.187263    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a64fae7c-510c-41a3-8a75-95203daac0bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a64fae7c-510c-41a3-8a75-95203daac0bf-maas-api-openshift-ca\") pod \"a64fae7c-510c-41a3-8a75-95203daac0bf\" (UID: \"a64fae7c-510c-41a3-8a75-95203daac0bf\") "
Nov 03 09:17:42.188228 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:42.187377    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a64fae7c-510c-41a3-8a75-95203daac0bf-kube-api-access-f5ptr\" (UniqueName: \"kubernetes.io/projected/a64fae7c-510c-41a3-8a75-95203daac0bf-kube-api-access-f5ptr\") pod \"a64fae7c-510c-41a3-8a75-95203daac0bf\" (UID: \"a64fae7c-510c-41a3-8a75-95203daac0bf\") "
Nov 03 09:17:42.254004 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:42.227659    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a64fae7c-510c-41a3-8a75-95203daac0bf-kube-api-access-f5ptr" pod "a64fae7c-510c-41a3-8a75-95203daac0bf" (UID: "a64fae7c-510c-41a3-8a75-95203daac0bf"). InnerVolumeSpecName "kube-api-access-f5ptr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:17:42.254004 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:42.230162    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a64fae7c-510c-41a3-8a75-95203daac0bf-maas-api-openshift-ca" pod "a64fae7c-510c-41a3-8a75-95203daac0bf" (UID: "a64fae7c-510c-41a3-8a75-95203daac0bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:17:42.280670 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:42.280574    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a64fae7c-510c-41a3-8a75-95203daac0bf-maas-api-xks-ca" pod "a64fae7c-510c-41a3-8a75-95203daac0bf" (UID: "a64fae7c-510c-41a3-8a75-95203daac0bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:17:42.288586 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:42.288554    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a64fae7c-510c-41a3-8a75-95203daac0bf-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:17:42.288586 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:42.288587    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5ptr\" (UniqueName: \"kubernetes.io/projected/a64fae7c-510c-41a3-8a75-95203daac0bf-kube-api-access-f5ptr\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:17:42.288818 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:42.288601    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a64fae7c-510c-41a3-8a75-95203daac0bf-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:17:42.930519 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:42.930487    4473 generic.go:356] "Generic (PLEG): container finished" podID="cfb9ba89-9bc5-4024-b535-deeb4421b4e2" containerID="dfbbb06c23abd7b62ea93ad06f2304b0544743612e2249fa7ad8287f6438725f" exitCode=0
Nov 03 09:17:42.930913 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:42.930571    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00db84b1-hxwqt" event={"ID":"cfb9ba89-9bc5-4024-b535-deeb4421b4e2","Type":"ContainerDied","Data":"dfbbb06c23abd7b62ea93ad06f2304b0544743612e2249fa7ad8287f6438725f"}
Nov 03 09:17:42.932037 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:42.932015    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7188c7f1-5x2jb" event={"ID":"a64fae7c-510c-41a3-8a75-95203daac0bf","Type":"ContainerDied","Data":"15ee67fb027a6e6aadb3dc460dabfeb41dc75899856abe4fdb632d3e4bf10a21"}
Nov 03 09:17:42.932037 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:42.932032    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7188c7f1-5x2jb"
Nov 03 09:17:42.932185 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:42.932056    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15ee67fb027a6e6aadb3dc460dabfeb41dc75899856abe4fdb632d3e4bf10a21"
Nov 03 09:17:44.082193 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:44.082169    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00db84b1-hxwqt"
Nov 03 09:17:44.208427 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:44.208347    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfb9ba89-9bc5-4024-b535-deeb4421b4e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfb9ba89-9bc5-4024-b535-deeb4421b4e2-maas-api-xks-ca\") pod \"cfb9ba89-9bc5-4024-b535-deeb4421b4e2\" (UID: \"cfb9ba89-9bc5-4024-b535-deeb4421b4e2\") "
Nov 03 09:17:44.208569 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:44.208452    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfb9ba89-9bc5-4024-b535-deeb4421b4e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfb9ba89-9bc5-4024-b535-deeb4421b4e2-maas-api-openshift-ca\") pod \"cfb9ba89-9bc5-4024-b535-deeb4421b4e2\" (UID: \"cfb9ba89-9bc5-4024-b535-deeb4421b4e2\") "
Nov 03 09:17:44.208569 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:44.208519    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfb9ba89-9bc5-4024-b535-deeb4421b4e2-kube-api-access-kzr99\" (UniqueName: \"kubernetes.io/projected/cfb9ba89-9bc5-4024-b535-deeb4421b4e2-kube-api-access-kzr99\") pod \"cfb9ba89-9bc5-4024-b535-deeb4421b4e2\" (UID: \"cfb9ba89-9bc5-4024-b535-deeb4421b4e2\") "
Nov 03 09:17:44.208868 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:44.208842    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfb9ba89-9bc5-4024-b535-deeb4421b4e2-maas-api-openshift-ca" pod "cfb9ba89-9bc5-4024-b535-deeb4421b4e2" (UID: "cfb9ba89-9bc5-4024-b535-deeb4421b4e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:17:44.210670 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:44.210642    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfb9ba89-9bc5-4024-b535-deeb4421b4e2-maas-api-xks-ca" pod "cfb9ba89-9bc5-4024-b535-deeb4421b4e2" (UID: "cfb9ba89-9bc5-4024-b535-deeb4421b4e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:17:44.211154 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:44.211133    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb9ba89-9bc5-4024-b535-deeb4421b4e2-kube-api-access-kzr99" pod "cfb9ba89-9bc5-4024-b535-deeb4421b4e2" (UID: "cfb9ba89-9bc5-4024-b535-deeb4421b4e2"). InnerVolumeSpecName "kube-api-access-kzr99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:17:44.310563 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:44.310534    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfb9ba89-9bc5-4024-b535-deeb4421b4e2-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:17:44.310563 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:44.310561    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfb9ba89-9bc5-4024-b535-deeb4421b4e2-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:17:44.310738 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:44.310571    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzr99\" (UniqueName: \"kubernetes.io/projected/cfb9ba89-9bc5-4024-b535-deeb4421b4e2-kube-api-access-kzr99\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:17:44.945523 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:44.945484    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00db84b1-hxwqt" event={"ID":"cfb9ba89-9bc5-4024-b535-deeb4421b4e2","Type":"ContainerDied","Data":"03137da55e601d4160245ef51c22858f4f888152cf3c3506de5a53fbb82db34d"}
Nov 03 09:17:44.945721 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:44.945539    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03137da55e601d4160245ef51c22858f4f888152cf3c3506de5a53fbb82db34d"
Nov 03 09:17:44.945721 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:44.945503    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00db84b1-hxwqt"
Nov 03 09:17:46.937636 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:17:46.937536    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:17:52.053793 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:52.053763    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38608a06-qmvgx"]
Nov 03 09:17:52.054276 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:52.054262    4473 cpu_manager.go:414] "containerMap: removing container" podUID="cfb9ba89-9bc5-4024-b535-deeb4421b4e2" containerName="revoke-keys"
Nov 03 09:17:52.054333 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:52.054277    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfb9ba89-9bc5-4024-b535-deeb4421b4e2" containerName="revoke-keys"
Nov 03 09:17:52.054333 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:52.054295    4473 cpu_manager.go:414] "containerMap: removing container" podUID="a64fae7c-510c-41a3-8a75-95203daac0bf" containerName="revoke-keys"
Nov 03 09:17:52.054333 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:52.054300    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a64fae7c-510c-41a3-8a75-95203daac0bf" containerName="revoke-keys"
Nov 03 09:17:52.054426 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:52.054392    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfb9ba89-9bc5-4024-b535-deeb4421b4e2" containerName="revoke-keys"
Nov 03 09:17:52.054426 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:52.054403    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="a64fae7c-510c-41a3-8a75-95203daac0bf" containerName="revoke-keys"
Nov 03 09:17:52.057830 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:52.057813    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38608a06-qmvgx"
Nov 03 09:17:52.117917 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:52.117884    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38608a06-qmvgx"]
Nov 03 09:17:52.158190 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:52.158160    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b0ba314-f331-4642-8690-5bbcb943f9ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38608a06-qmvgx\" (UID: \"5b0ba314-f331-4642-8690-5bbcb943f9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38608a06-qmvgx"
Nov 03 09:17:52.158391 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:52.158369    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch5h4\" (UniqueName: \"kubernetes.io/projected/5b0ba314-f331-4642-8690-5bbcb943f9ed-kube-api-access-ch5h4\") pod \"maas-api-revoke-sub-38608a06-qmvgx\" (UID: \"5b0ba314-f331-4642-8690-5bbcb943f9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38608a06-qmvgx"
Nov 03 09:17:52.158489 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:52.158418    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b0ba314-f331-4642-8690-5bbcb943f9ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38608a06-qmvgx\" (UID: \"5b0ba314-f331-4642-8690-5bbcb943f9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38608a06-qmvgx"
Nov 03 09:17:52.162850 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:52.162828    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:17:52.162974 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:52.162887    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:17:52.162974 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:52.162888    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:17:52.162974 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:52.162922    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:17:52.260199 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:52.260168    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch5h4\" (UniqueName: \"kubernetes.io/projected/5b0ba314-f331-4642-8690-5bbcb943f9ed-kube-api-access-ch5h4\") pod \"maas-api-revoke-sub-38608a06-qmvgx\" (UID: \"5b0ba314-f331-4642-8690-5bbcb943f9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38608a06-qmvgx"
Nov 03 09:17:52.260380 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:52.260207    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b0ba314-f331-4642-8690-5bbcb943f9ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38608a06-qmvgx\" (UID: \"5b0ba314-f331-4642-8690-5bbcb943f9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38608a06-qmvgx"
Nov 03 09:17:52.260380 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:52.260294    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b0ba314-f331-4642-8690-5bbcb943f9ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38608a06-qmvgx\" (UID: \"5b0ba314-f331-4642-8690-5bbcb943f9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38608a06-qmvgx"
Nov 03 09:17:52.261129 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:52.261092    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b0ba314-f331-4642-8690-5bbcb943f9ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38608a06-qmvgx\" (UID: \"5b0ba314-f331-4642-8690-5bbcb943f9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38608a06-qmvgx"
Nov 03 09:17:52.262829 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:52.262813    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b0ba314-f331-4642-8690-5bbcb943f9ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38608a06-qmvgx\" (UID: \"5b0ba314-f331-4642-8690-5bbcb943f9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38608a06-qmvgx"
Nov 03 09:17:52.320320 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:52.320246    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch5h4\" (UniqueName: \"kubernetes.io/projected/5b0ba314-f331-4642-8690-5bbcb943f9ed-kube-api-access-ch5h4\") pod \"maas-api-revoke-sub-38608a06-qmvgx\" (UID: \"5b0ba314-f331-4642-8690-5bbcb943f9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38608a06-qmvgx"
Nov 03 09:17:52.371341 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:52.371311    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38608a06-qmvgx"
Nov 03 09:17:52.754427 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:52.754405    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38608a06-qmvgx"]
Nov 03 09:17:52.755100 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:17:52.755078    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0ba314_f331_4642_8690_5bbcb943f9ed.slice/crio-9e02701fb053c4093a1b7248ecdf4a5519e8e870b24a36325aef301b8dddb072 WatchSource:0}: Error finding container 9e02701fb053c4093a1b7248ecdf4a5519e8e870b24a36325aef301b8dddb072: Status 404 returned error can't find the container with id 9e02701fb053c4093a1b7248ecdf4a5519e8e870b24a36325aef301b8dddb072
Nov 03 09:17:52.999459 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:52.999426    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38608a06-qmvgx" event={"ID":"5b0ba314-f331-4642-8690-5bbcb943f9ed","Type":"ContainerStarted","Data":"9e02701fb053c4093a1b7248ecdf4a5519e8e870b24a36325aef301b8dddb072"}
Nov 03 09:17:54.007386 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:54.007295    4473 generic.go:356] "Generic (PLEG): container finished" podID="5b0ba314-f331-4642-8690-5bbcb943f9ed" containerID="3290c3fcc4d84430a2aad5d4599b547a624cab6c61b4e3a780e8ceb25f98785a" exitCode=0
Nov 03 09:17:54.007386 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:54.007336    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38608a06-qmvgx" event={"ID":"5b0ba314-f331-4642-8690-5bbcb943f9ed","Type":"ContainerDied","Data":"3290c3fcc4d84430a2aad5d4599b547a624cab6c61b4e3a780e8ceb25f98785a"}
Nov 03 09:17:55.249034 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:55.249008    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38608a06-qmvgx"
Nov 03 09:17:55.294013 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:55.293917    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b0ba314-f331-4642-8690-5bbcb943f9ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b0ba314-f331-4642-8690-5bbcb943f9ed-maas-api-openshift-ca\") pod \"5b0ba314-f331-4642-8690-5bbcb943f9ed\" (UID: \"5b0ba314-f331-4642-8690-5bbcb943f9ed\") "
Nov 03 09:17:55.294156 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:55.294053    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b0ba314-f331-4642-8690-5bbcb943f9ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b0ba314-f331-4642-8690-5bbcb943f9ed-maas-api-xks-ca\") pod \"5b0ba314-f331-4642-8690-5bbcb943f9ed\" (UID: \"5b0ba314-f331-4642-8690-5bbcb943f9ed\") "
Nov 03 09:17:55.294156 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:55.294115    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b0ba314-f331-4642-8690-5bbcb943f9ed-kube-api-access-ch5h4\" (UniqueName: \"kubernetes.io/projected/5b0ba314-f331-4642-8690-5bbcb943f9ed-kube-api-access-ch5h4\") pod \"5b0ba314-f331-4642-8690-5bbcb943f9ed\" (UID: \"5b0ba314-f331-4642-8690-5bbcb943f9ed\") "
Nov 03 09:17:55.294332 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:55.294311    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b0ba314-f331-4642-8690-5bbcb943f9ed-maas-api-openshift-ca" pod "5b0ba314-f331-4642-8690-5bbcb943f9ed" (UID: "5b0ba314-f331-4642-8690-5bbcb943f9ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:17:55.294571 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:55.294557    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b0ba314-f331-4642-8690-5bbcb943f9ed-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:17:55.296538 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:55.296498    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b0ba314-f331-4642-8690-5bbcb943f9ed-maas-api-xks-ca" pod "5b0ba314-f331-4642-8690-5bbcb943f9ed" (UID: "5b0ba314-f331-4642-8690-5bbcb943f9ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:17:55.296810 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:55.296778    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b0ba314-f331-4642-8690-5bbcb943f9ed-kube-api-access-ch5h4" pod "5b0ba314-f331-4642-8690-5bbcb943f9ed" (UID: "5b0ba314-f331-4642-8690-5bbcb943f9ed"). InnerVolumeSpecName "kube-api-access-ch5h4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:17:55.395576 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:55.395542    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b0ba314-f331-4642-8690-5bbcb943f9ed-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:17:55.395576 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:55.395581    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch5h4\" (UniqueName: \"kubernetes.io/projected/5b0ba314-f331-4642-8690-5bbcb943f9ed-kube-api-access-ch5h4\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:17:56.021283 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:56.021251    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38608a06-qmvgx" event={"ID":"5b0ba314-f331-4642-8690-5bbcb943f9ed","Type":"ContainerDied","Data":"9e02701fb053c4093a1b7248ecdf4a5519e8e870b24a36325aef301b8dddb072"}
Nov 03 09:17:56.021283 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:56.021276    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38608a06-qmvgx"
Nov 03 09:17:56.021488 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:17:56.021297    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e02701fb053c4093a1b7248ecdf4a5519e8e870b24a36325aef301b8dddb072"
Nov 03 09:18:02.118313 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:02.118281    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:18:09.509977 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:09.509720    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"]
Nov 03 09:18:09.515536 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:09.510542    4473 cpu_manager.go:414] "containerMap: removing container" podUID="5b0ba314-f331-4642-8690-5bbcb943f9ed" containerName="revoke-keys"
Nov 03 09:18:09.515536 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:09.510568    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b0ba314-f331-4642-8690-5bbcb943f9ed" containerName="revoke-keys"
Nov 03 09:18:09.515536 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:09.510696    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b0ba314-f331-4642-8690-5bbcb943f9ed" containerName="revoke-keys"
Nov 03 09:18:09.526381 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:09.526351    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"
Nov 03 09:18:09.558756 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:09.558729    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"]
Nov 03 09:18:09.641229 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:09.641194    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63a764c9-6c46-47e6-a12a-190d5d4707a8-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq\" (UID: \"63a764c9-6c46-47e6-a12a-190d5d4707a8\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"
Nov 03 09:18:09.641400 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:09.641263    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63a764c9-6c46-47e6-a12a-190d5d4707a8-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq\" (UID: \"63a764c9-6c46-47e6-a12a-190d5d4707a8\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"
Nov 03 09:18:09.641400 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:09.641337    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63a764c9-6c46-47e6-a12a-190d5d4707a8-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq\" (UID: \"63a764c9-6c46-47e6-a12a-190d5d4707a8\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"
Nov 03 09:18:09.641400 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:09.641384    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63a764c9-6c46-47e6-a12a-190d5d4707a8-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq\" (UID: \"63a764c9-6c46-47e6-a12a-190d5d4707a8\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"
Nov 03 09:18:09.641503 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:09.641410    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63a764c9-6c46-47e6-a12a-190d5d4707a8-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq\" (UID: \"63a764c9-6c46-47e6-a12a-190d5d4707a8\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"
Nov 03 09:18:09.641543 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:09.641508    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63a764c9-6c46-47e6-a12a-190d5d4707a8-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq\" (UID: \"63a764c9-6c46-47e6-a12a-190d5d4707a8\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"
Nov 03 09:18:09.641631 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:09.641618    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9st4\" (UniqueName: \"kubernetes.io/projected/63a764c9-6c46-47e6-a12a-190d5d4707a8-kube-api-access-x9st4\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq\" (UID: \"63a764c9-6c46-47e6-a12a-190d5d4707a8\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"
Nov 03 09:18:09.641667 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:09.641657    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63a764c9-6c46-47e6-a12a-190d5d4707a8-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq\" (UID: \"63a764c9-6c46-47e6-a12a-190d5d4707a8\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"
Nov 03 09:18:09.641726 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:09.641710    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63a764c9-6c46-47e6-a12a-190d5d4707a8-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq\" (UID: \"63a764c9-6c46-47e6-a12a-190d5d4707a8\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"
Nov 03 09:18:09.743201 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:09.743167    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63a764c9-6c46-47e6-a12a-190d5d4707a8-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq\" (UID: \"63a764c9-6c46-47e6-a12a-190d5d4707a8\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"
Nov 03 09:18:09.743363 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:09.743254    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63a764c9-6c46-47e6-a12a-190d5d4707a8-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq\" (UID: \"63a764c9-6c46-47e6-a12a-190d5d4707a8\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"
Nov 03 09:18:09.743363 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:09.743284    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63a764c9-6c46-47e6-a12a-190d5d4707a8-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq\" (UID: \"63a764c9-6c46-47e6-a12a-190d5d4707a8\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"
Nov 03 09:18:09.743363 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:09.743323    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63a764c9-6c46-47e6-a12a-190d5d4707a8-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq\" (UID: \"63a764c9-6c46-47e6-a12a-190d5d4707a8\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"
Nov 03 09:18:09.743513 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:09.743359    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63a764c9-6c46-47e6-a12a-190d5d4707a8-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq\" (UID: \"63a764c9-6c46-47e6-a12a-190d5d4707a8\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"
Nov 03 09:18:09.743513 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:09.743390    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63a764c9-6c46-47e6-a12a-190d5d4707a8-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq\" (UID: \"63a764c9-6c46-47e6-a12a-190d5d4707a8\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"
Nov 03 09:18:09.743513 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:09.743471    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9st4\" (UniqueName: \"kubernetes.io/projected/63a764c9-6c46-47e6-a12a-190d5d4707a8-kube-api-access-x9st4\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq\" (UID: \"63a764c9-6c46-47e6-a12a-190d5d4707a8\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"
Nov 03 09:18:09.743673 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:09.743516    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63a764c9-6c46-47e6-a12a-190d5d4707a8-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq\" (UID: \"63a764c9-6c46-47e6-a12a-190d5d4707a8\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"
Nov 03 09:18:09.743673 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:09.743551    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63a764c9-6c46-47e6-a12a-190d5d4707a8-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq\" (UID: \"63a764c9-6c46-47e6-a12a-190d5d4707a8\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"
Nov 03 09:18:09.743810 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:09.743733    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63a764c9-6c46-47e6-a12a-190d5d4707a8-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq\" (UID: \"63a764c9-6c46-47e6-a12a-190d5d4707a8\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"
Nov 03 09:18:09.743810 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:09.743782    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63a764c9-6c46-47e6-a12a-190d5d4707a8-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq\" (UID: \"63a764c9-6c46-47e6-a12a-190d5d4707a8\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"
Nov 03 09:18:09.743979 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:09.743851    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63a764c9-6c46-47e6-a12a-190d5d4707a8-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq\" (UID: \"63a764c9-6c46-47e6-a12a-190d5d4707a8\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"
Nov 03 09:18:09.744045 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:09.744031    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63a764c9-6c46-47e6-a12a-190d5d4707a8-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq\" (UID: \"63a764c9-6c46-47e6-a12a-190d5d4707a8\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"
Nov 03 09:18:09.744589 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:09.744566    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63a764c9-6c46-47e6-a12a-190d5d4707a8-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq\" (UID: \"63a764c9-6c46-47e6-a12a-190d5d4707a8\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"
Nov 03 09:18:09.745772 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:09.745750    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63a764c9-6c46-47e6-a12a-190d5d4707a8-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq\" (UID: \"63a764c9-6c46-47e6-a12a-190d5d4707a8\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"
Nov 03 09:18:09.746174 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:09.746155    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63a764c9-6c46-47e6-a12a-190d5d4707a8-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq\" (UID: \"63a764c9-6c46-47e6-a12a-190d5d4707a8\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"
Nov 03 09:18:09.802570 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:09.802512    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9st4\" (UniqueName: \"kubernetes.io/projected/63a764c9-6c46-47e6-a12a-190d5d4707a8-kube-api-access-x9st4\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq\" (UID: \"63a764c9-6c46-47e6-a12a-190d5d4707a8\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"
Nov 03 09:18:09.802657 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:09.802583    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63a764c9-6c46-47e6-a12a-190d5d4707a8-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq\" (UID: \"63a764c9-6c46-47e6-a12a-190d5d4707a8\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"
Nov 03 09:18:09.841730 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:09.841705    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"
Nov 03 09:18:10.275121 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:10.275090    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"]
Nov 03 09:18:10.283984 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:10.280677    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:18:10.283984 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:10.280849    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:18:10.283984 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:10.280990    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:18:11.178761 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:11.178725    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq" event={"ID":"63a764c9-6c46-47e6-a12a-190d5d4707a8","Type":"ContainerStarted","Data":"706e97692add275f449d0f5ece8eefb2e5c3277802c9ac887a63ea442e93e1c7"}
Nov 03 09:18:11.179211 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:11.178819    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq" event={"ID":"63a764c9-6c46-47e6-a12a-190d5d4707a8","Type":"ContainerStarted","Data":"e4f320287512ed2cdbf0059f653d903545f9f8f8717424d1cc32eba2040a8be1"}
Nov 03 09:18:11.350553 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:11.350505    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq" podStartSLOduration=2.350490497 podStartE2EDuration="2.350490497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:18:11.346628907 +0000 UTC m=+2315.615692383" watchObservedRunningTime="2026-11-03 09:18:11.350490497 +0000 UTC m=+2315.619553931"
Nov 03 09:18:11.842422 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:11.842368    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"
Nov 03 09:18:11.844117 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:11.844086    4473 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 03 09:18:11.844263 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:11.844161    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq" podUID="63a764c9-6c46-47e6-a12a-190d5d4707a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 03 09:18:12.842316 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:12.842281    4473 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 03 09:18:12.842844 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:12.842332    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq" podUID="63a764c9-6c46-47e6-a12a-190d5d4707a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 03 09:18:13.842408 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:13.842361    4473 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 03 09:18:13.842908 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:13.842424    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq" podUID="63a764c9-6c46-47e6-a12a-190d5d4707a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 03 09:18:14.842448 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:14.842407    4473 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 03 09:18:14.842859 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:14.842471    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq" podUID="63a764c9-6c46-47e6-a12a-190d5d4707a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 03 09:18:15.842968 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:15.842545    4473 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 03 09:18:15.842968 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:15.842615    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq" podUID="63a764c9-6c46-47e6-a12a-190d5d4707a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 03 09:18:16.842828 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:16.842788    4473 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 03 09:18:16.843023 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:16.842859    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq" podUID="63a764c9-6c46-47e6-a12a-190d5d4707a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 03 09:18:16.938013 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:18:16.937968    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:18:17.893850 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:17.893816    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"
Nov 03 09:18:17.894353 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:17.894320    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"
Nov 03 09:18:17.895071 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:17.895051    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"
Nov 03 09:18:33.283552 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:33.283518    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dce7c497-rr48n"]
Nov 03 09:18:33.287426 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:33.287412    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce7c497-rr48n"
Nov 03 09:18:33.344545 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:33.344519    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dce7c497-rr48n"]
Nov 03 09:18:33.357731 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:33.357705    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c5d409a-585f-4947-a600-7c504df3321e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dce7c497-rr48n\" (UID: \"4c5d409a-585f-4947-a600-7c504df3321e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce7c497-rr48n"
Nov 03 09:18:33.357875 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:33.357766    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c5d409a-585f-4947-a600-7c504df3321e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dce7c497-rr48n\" (UID: \"4c5d409a-585f-4947-a600-7c504df3321e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce7c497-rr48n"
Nov 03 09:18:33.357875 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:33.357836    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2phjc\" (UniqueName: \"kubernetes.io/projected/4c5d409a-585f-4947-a600-7c504df3321e-kube-api-access-2phjc\") pod \"maas-api-revoke-sub-dce7c497-rr48n\" (UID: \"4c5d409a-585f-4947-a600-7c504df3321e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce7c497-rr48n"
Nov 03 09:18:33.390436 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:33.390414    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:18:33.390608 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:33.390515    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:18:33.391345 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:33.391332    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:18:33.391418 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:33.391403    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:18:33.459286 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:33.459260    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c5d409a-585f-4947-a600-7c504df3321e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dce7c497-rr48n\" (UID: \"4c5d409a-585f-4947-a600-7c504df3321e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce7c497-rr48n"
Nov 03 09:18:33.459483 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:33.459321    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c5d409a-585f-4947-a600-7c504df3321e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dce7c497-rr48n\" (UID: \"4c5d409a-585f-4947-a600-7c504df3321e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce7c497-rr48n"
Nov 03 09:18:33.459483 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:33.459367    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2phjc\" (UniqueName: \"kubernetes.io/projected/4c5d409a-585f-4947-a600-7c504df3321e-kube-api-access-2phjc\") pod \"maas-api-revoke-sub-dce7c497-rr48n\" (UID: \"4c5d409a-585f-4947-a600-7c504df3321e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce7c497-rr48n"
Nov 03 09:18:33.459982 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:33.459937    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c5d409a-585f-4947-a600-7c504df3321e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dce7c497-rr48n\" (UID: \"4c5d409a-585f-4947-a600-7c504df3321e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce7c497-rr48n"
Nov 03 09:18:33.461780 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:33.461756    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c5d409a-585f-4947-a600-7c504df3321e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dce7c497-rr48n\" (UID: \"4c5d409a-585f-4947-a600-7c504df3321e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce7c497-rr48n"
Nov 03 09:18:33.519165 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:33.519140    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2phjc\" (UniqueName: \"kubernetes.io/projected/4c5d409a-585f-4947-a600-7c504df3321e-kube-api-access-2phjc\") pod \"maas-api-revoke-sub-dce7c497-rr48n\" (UID: \"4c5d409a-585f-4947-a600-7c504df3321e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce7c497-rr48n"
Nov 03 09:18:33.600739 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:33.600660    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce7c497-rr48n"
Nov 03 09:18:33.978936 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:18:33.978908    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c5d409a_585f_4947_a600_7c504df3321e.slice/crio-036e4204a3f8438aa16b78acfcefc34b0317ce4db59f03545529401000419a42 WatchSource:0}: Error finding container 036e4204a3f8438aa16b78acfcefc34b0317ce4db59f03545529401000419a42: Status 404 returned error can't find the container with id 036e4204a3f8438aa16b78acfcefc34b0317ce4db59f03545529401000419a42
Nov 03 09:18:33.979086 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:33.979037    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dce7c497-rr48n"]
Nov 03 09:18:34.395722 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:34.395686    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce7c497-rr48n" event={"ID":"4c5d409a-585f-4947-a600-7c504df3321e","Type":"ContainerStarted","Data":"309c6dc8e1062ae5edbbfdf0c6d0c3cf6e087712079c15d99328851c8c992d15"}
Nov 03 09:18:34.396158 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:34.395750    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce7c497-rr48n" event={"ID":"4c5d409a-585f-4947-a600-7c504df3321e","Type":"ContainerStarted","Data":"036e4204a3f8438aa16b78acfcefc34b0317ce4db59f03545529401000419a42"}
Nov 03 09:18:34.516340 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:34.516244    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce7c497-rr48n" podStartSLOduration=1.516228597 podStartE2EDuration="1.516228597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:18:34.512617622 +0000 UTC m=+2338.781681069" watchObservedRunningTime="2026-11-03 09:18:34.516228597 +0000 UTC m=+2338.785292039"
Nov 03 09:18:35.403286 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:35.403254    4473 generic.go:356] "Generic (PLEG): container finished" podID="4c5d409a-585f-4947-a600-7c504df3321e" containerID="309c6dc8e1062ae5edbbfdf0c6d0c3cf6e087712079c15d99328851c8c992d15" exitCode=0
Nov 03 09:18:35.403701 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:35.403339    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce7c497-rr48n" event={"ID":"4c5d409a-585f-4947-a600-7c504df3321e","Type":"ContainerDied","Data":"309c6dc8e1062ae5edbbfdf0c6d0c3cf6e087712079c15d99328851c8c992d15"}
Nov 03 09:18:36.559192 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:36.559169    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce7c497-rr48n"
Nov 03 09:18:36.697174 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:36.697086    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c5d409a-585f-4947-a600-7c504df3321e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c5d409a-585f-4947-a600-7c504df3321e-maas-api-openshift-ca\") pod \"4c5d409a-585f-4947-a600-7c504df3321e\" (UID: \"4c5d409a-585f-4947-a600-7c504df3321e\") "
Nov 03 09:18:36.697174 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:36.697159    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c5d409a-585f-4947-a600-7c504df3321e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c5d409a-585f-4947-a600-7c504df3321e-maas-api-xks-ca\") pod \"4c5d409a-585f-4947-a600-7c504df3321e\" (UID: \"4c5d409a-585f-4947-a600-7c504df3321e\") "
Nov 03 09:18:36.697419 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:36.697195    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c5d409a-585f-4947-a600-7c504df3321e-kube-api-access-2phjc\" (UniqueName: \"kubernetes.io/projected/4c5d409a-585f-4947-a600-7c504df3321e-kube-api-access-2phjc\") pod \"4c5d409a-585f-4947-a600-7c504df3321e\" (UID: \"4c5d409a-585f-4947-a600-7c504df3321e\") "
Nov 03 09:18:36.697536 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:36.697510    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c5d409a-585f-4947-a600-7c504df3321e-maas-api-openshift-ca" pod "4c5d409a-585f-4947-a600-7c504df3321e" (UID: "4c5d409a-585f-4947-a600-7c504df3321e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:18:36.697694 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:36.697675    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c5d409a-585f-4947-a600-7c504df3321e-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:18:36.700020 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:36.699995    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c5d409a-585f-4947-a600-7c504df3321e-maas-api-xks-ca" pod "4c5d409a-585f-4947-a600-7c504df3321e" (UID: "4c5d409a-585f-4947-a600-7c504df3321e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:18:36.700159 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:36.700032    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c5d409a-585f-4947-a600-7c504df3321e-kube-api-access-2phjc" pod "4c5d409a-585f-4947-a600-7c504df3321e" (UID: "4c5d409a-585f-4947-a600-7c504df3321e"). InnerVolumeSpecName "kube-api-access-2phjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:18:36.798813 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:36.798785    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2phjc\" (UniqueName: \"kubernetes.io/projected/4c5d409a-585f-4947-a600-7c504df3321e-kube-api-access-2phjc\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:18:36.798813 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:36.798813    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c5d409a-585f-4947-a600-7c504df3321e-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:18:37.417786 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:37.417756    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce7c497-rr48n"
Nov 03 09:18:37.417979 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:37.417784    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dce7c497-rr48n" event={"ID":"4c5d409a-585f-4947-a600-7c504df3321e","Type":"ContainerDied","Data":"036e4204a3f8438aa16b78acfcefc34b0317ce4db59f03545529401000419a42"}
Nov 03 09:18:37.417979 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:37.417834    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="036e4204a3f8438aa16b78acfcefc34b0317ce4db59f03545529401000419a42"
Nov 03 09:18:43.032367 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:43.032335    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:18:45.731162 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:45.731126    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"]
Nov 03 09:18:45.736399 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:45.736373    4473 cpu_manager.go:414] "containerMap: removing container" podUID="4c5d409a-585f-4947-a600-7c504df3321e" containerName="revoke-keys"
Nov 03 09:18:45.736562 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:45.736549    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c5d409a-585f-4947-a600-7c504df3321e" containerName="revoke-keys"
Nov 03 09:18:45.736792 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:45.736779    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c5d409a-585f-4947-a600-7c504df3321e" containerName="revoke-keys"
Nov 03 09:18:45.743245 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:45.743225    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"
Nov 03 09:18:45.746759 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:45.746739    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"]
Nov 03 09:18:45.765498 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:45.765465    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/909a4fb7-462b-4d1b-a96f-8da66cb52d11-istio-data\") pod \"e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd\" (UID: \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\") " pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"
Nov 03 09:18:45.765731 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:45.765712    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/909a4fb7-462b-4d1b-a96f-8da66cb52d11-credential-socket\") pod \"e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd\" (UID: \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\") " pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"
Nov 03 09:18:45.765901 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:45.765886    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/909a4fb7-462b-4d1b-a96f-8da66cb52d11-istio-envoy\") pod \"e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd\" (UID: \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\") " pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"
Nov 03 09:18:45.766149 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:45.766132    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/909a4fb7-462b-4d1b-a96f-8da66cb52d11-istiod-ca-cert\") pod \"e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd\" (UID: \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\") " pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"
Nov 03 09:18:45.766355 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:45.766336    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzsgn\" (UniqueName: \"kubernetes.io/projected/909a4fb7-462b-4d1b-a96f-8da66cb52d11-kube-api-access-mzsgn\") pod \"e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd\" (UID: \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\") " pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"
Nov 03 09:18:45.766538 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:45.766518    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/909a4fb7-462b-4d1b-a96f-8da66cb52d11-istio-token\") pod \"e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd\" (UID: \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\") " pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"
Nov 03 09:18:45.766735 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:45.766715    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/909a4fb7-462b-4d1b-a96f-8da66cb52d11-workload-socket\") pod \"e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd\" (UID: \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\") " pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"
Nov 03 09:18:45.766895 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:45.766880    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/909a4fb7-462b-4d1b-a96f-8da66cb52d11-istio-podinfo\") pod \"e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd\" (UID: \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\") " pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"
Nov 03 09:18:45.767069 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:45.767053    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/909a4fb7-462b-4d1b-a96f-8da66cb52d11-workload-certs\") pod \"e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd\" (UID: \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\") " pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"
Nov 03 09:18:45.867807 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:45.867755    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/909a4fb7-462b-4d1b-a96f-8da66cb52d11-credential-socket\") pod \"e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd\" (UID: \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\") " pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"
Nov 03 09:18:45.867807 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:45.867810    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/909a4fb7-462b-4d1b-a96f-8da66cb52d11-istio-envoy\") pod \"e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd\" (UID: \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\") " pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"
Nov 03 09:18:45.868124 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:45.867838    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/909a4fb7-462b-4d1b-a96f-8da66cb52d11-istiod-ca-cert\") pod \"e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd\" (UID: \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\") " pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"
Nov 03 09:18:45.868124 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:45.867938    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzsgn\" (UniqueName: \"kubernetes.io/projected/909a4fb7-462b-4d1b-a96f-8da66cb52d11-kube-api-access-mzsgn\") pod \"e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd\" (UID: \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\") " pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"
Nov 03 09:18:45.868124 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:45.868025    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/909a4fb7-462b-4d1b-a96f-8da66cb52d11-istio-token\") pod \"e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd\" (UID: \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\") " pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"
Nov 03 09:18:45.868124 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:45.868066    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/909a4fb7-462b-4d1b-a96f-8da66cb52d11-workload-socket\") pod \"e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd\" (UID: \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\") " pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"
Nov 03 09:18:45.868124 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:45.868096    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/909a4fb7-462b-4d1b-a96f-8da66cb52d11-istio-podinfo\") pod \"e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd\" (UID: \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\") " pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"
Nov 03 09:18:45.868124 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:45.868122    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/909a4fb7-462b-4d1b-a96f-8da66cb52d11-workload-certs\") pod \"e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd\" (UID: \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\") " pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"
Nov 03 09:18:45.868459 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:45.868271    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/909a4fb7-462b-4d1b-a96f-8da66cb52d11-istio-data\") pod \"e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd\" (UID: \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\") " pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"
Nov 03 09:18:45.869089 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:45.869059    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/909a4fb7-462b-4d1b-a96f-8da66cb52d11-workload-socket\") pod \"e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd\" (UID: \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\") " pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"
Nov 03 09:18:45.869391 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:45.869354    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/909a4fb7-462b-4d1b-a96f-8da66cb52d11-istiod-ca-cert\") pod \"e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd\" (UID: \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\") " pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"
Nov 03 09:18:45.870361 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:45.870339    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/909a4fb7-462b-4d1b-a96f-8da66cb52d11-credential-socket\") pod \"e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd\" (UID: \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\") " pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"
Nov 03 09:18:45.870494 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:45.870447    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/909a4fb7-462b-4d1b-a96f-8da66cb52d11-istio-data\") pod \"e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd\" (UID: \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\") " pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"
Nov 03 09:18:45.870589 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:45.870510    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/909a4fb7-462b-4d1b-a96f-8da66cb52d11-workload-certs\") pod \"e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd\" (UID: \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\") " pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"
Nov 03 09:18:45.872913 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:45.872889    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/909a4fb7-462b-4d1b-a96f-8da66cb52d11-istio-envoy\") pod \"e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd\" (UID: \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\") " pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"
Nov 03 09:18:45.873641 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:45.873621    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/909a4fb7-462b-4d1b-a96f-8da66cb52d11-istio-podinfo\") pod \"e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd\" (UID: \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\") " pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"
Nov 03 09:18:45.879440 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:45.879418    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-f9b517-openshift-default-dockercfg-8mn4x\""
Nov 03 09:18:45.931069 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:45.931035    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/909a4fb7-462b-4d1b-a96f-8da66cb52d11-istio-token\") pod \"e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd\" (UID: \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\") " pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"
Nov 03 09:18:45.976120 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:45.976091    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzsgn\" (UniqueName: \"kubernetes.io/projected/909a4fb7-462b-4d1b-a96f-8da66cb52d11-kube-api-access-mzsgn\") pod \"e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd\" (UID: \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\") " pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"
Nov 03 09:18:46.065217 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:46.065145    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"
Nov 03 09:18:46.496298 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:46.496267    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"]
Nov 03 09:18:46.496418 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:18:46.496299    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod909a4fb7_462b_4d1b_a96f_8da66cb52d11.slice/crio-3b9c1e417091725a8d7a558e6d50b97e78ae2b713d93e2e781e725c844b148f6 WatchSource:0}: Error finding container 3b9c1e417091725a8d7a558e6d50b97e78ae2b713d93e2e781e725c844b148f6: Status 404 returned error can't find the container with id 3b9c1e417091725a8d7a558e6d50b97e78ae2b713d93e2e781e725c844b148f6
Nov 03 09:18:46.498815 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:46.498786    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:18:46.498892 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:46.498844    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:18:46.498892 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:46.498876    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:18:46.644729 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:46.644673    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd" event={"ID":"909a4fb7-462b-4d1b-a96f-8da66cb52d11","Type":"ContainerStarted","Data":"0ecc964583511d973a3acbc4c4e2030887d6841e0d68f884b85cef3bb1e5c8ce"}
Nov 03 09:18:46.644914 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:46.644753    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd" event={"ID":"909a4fb7-462b-4d1b-a96f-8da66cb52d11","Type":"ContainerStarted","Data":"3b9c1e417091725a8d7a558e6d50b97e78ae2b713d93e2e781e725c844b148f6"}
Nov 03 09:18:46.768336 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:46.768196    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd" podStartSLOduration=1.7681762559999998 podStartE2EDuration="1.768176256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:18:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:18:46.764245877 +0000 UTC m=+2351.033309309" watchObservedRunningTime="2026-11-03 09:18:46.768176256 +0000 UTC m=+2351.037239695"
Nov 03 09:18:46.973859 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:18:46.973829    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:18:47.066067 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:47.065986    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"
Nov 03 09:18:47.071330 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:47.071308    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"
Nov 03 09:18:47.651762 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:47.651733    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"
Nov 03 09:18:47.652849 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:47.652828    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"
Nov 03 09:18:54.441022 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:54.440988    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f9b517-646c9dd495-qt8s8"]
Nov 03 09:18:54.444915 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:54.444896    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f9b517-646c9dd495-qt8s8"
Nov 03 09:18:54.502984 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:54.502926    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f9b517-646c9dd495-qt8s8"]
Nov 03 09:18:54.522084 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:54.521470    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/842a3cc8-4062-41df-aa2f-18c19bb342c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-f9b517-646c9dd495-qt8s8\" (UID: \"842a3cc8-4062-41df-aa2f-18c19bb342c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f9b517-646c9dd495-qt8s8"
Nov 03 09:18:54.522084 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:54.521611    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpzls\" (UniqueName: \"kubernetes.io/projected/842a3cc8-4062-41df-aa2f-18c19bb342c9-kube-api-access-cpzls\") pod \"payload-pre-processing-e2e-shared-a-w0-f9b517-646c9dd495-qt8s8\" (UID: \"842a3cc8-4062-41df-aa2f-18c19bb342c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f9b517-646c9dd495-qt8s8"
Nov 03 09:18:54.560967 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:54.550376    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-f9b517-dockercfg-gxv9j\""
Nov 03 09:18:54.560967 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:54.550535    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-f9b517\""
Nov 03 09:18:54.652275 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:54.652241    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpzls\" (UniqueName: \"kubernetes.io/projected/842a3cc8-4062-41df-aa2f-18c19bb342c9-kube-api-access-cpzls\") pod \"payload-pre-processing-e2e-shared-a-w0-f9b517-646c9dd495-qt8s8\" (UID: \"842a3cc8-4062-41df-aa2f-18c19bb342c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f9b517-646c9dd495-qt8s8"
Nov 03 09:18:54.652529 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:54.652450    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/842a3cc8-4062-41df-aa2f-18c19bb342c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-f9b517-646c9dd495-qt8s8\" (UID: \"842a3cc8-4062-41df-aa2f-18c19bb342c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f9b517-646c9dd495-qt8s8"
Nov 03 09:18:54.666567 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:54.661518    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/842a3cc8-4062-41df-aa2f-18c19bb342c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-f9b517-646c9dd495-qt8s8\" (UID: \"842a3cc8-4062-41df-aa2f-18c19bb342c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f9b517-646c9dd495-qt8s8"
Nov 03 09:18:54.666567 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:54.661599    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-f9b517-6f6ff64d89-qd7zj"]
Nov 03 09:18:54.666809 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:54.666666    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f9b517-6f6ff64d89-qd7zj"
Nov 03 09:18:54.682751 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:54.682718    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-f9b517-6f6ff64d89-qd7zj"]
Nov 03 09:18:54.711219 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:54.711152    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpzls\" (UniqueName: \"kubernetes.io/projected/842a3cc8-4062-41df-aa2f-18c19bb342c9-kube-api-access-cpzls\") pod \"payload-pre-processing-e2e-shared-a-w0-f9b517-646c9dd495-qt8s8\" (UID: \"842a3cc8-4062-41df-aa2f-18c19bb342c9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f9b517-646c9dd495-qt8s8"
Nov 03 09:18:54.754180 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:54.754144    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-f9b517-6f6ff64d89-qd7zj\" (UID: \"c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f9b517-6f6ff64d89-qd7zj"
Nov 03 09:18:54.754180 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:54.754188    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpjnw\" (UniqueName: \"kubernetes.io/projected/c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9-kube-api-access-lpjnw\") pod \"payload-processing-e2e-shared-a-w0-f9b517-6f6ff64d89-qd7zj\" (UID: \"c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f9b517-6f6ff64d89-qd7zj"
Nov 03 09:18:54.760769 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:54.760739    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f9b517-646c9dd495-qt8s8"
Nov 03 09:18:54.855840 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:54.855782    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-f9b517-6f6ff64d89-qd7zj\" (UID: \"c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f9b517-6f6ff64d89-qd7zj"
Nov 03 09:18:54.856029 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:54.855858    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpjnw\" (UniqueName: \"kubernetes.io/projected/c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9-kube-api-access-lpjnw\") pod \"payload-processing-e2e-shared-a-w0-f9b517-6f6ff64d89-qd7zj\" (UID: \"c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f9b517-6f6ff64d89-qd7zj"
Nov 03 09:18:54.856916 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:54.856896    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-f9b517-6f6ff64d89-qd7zj\" (UID: \"c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f9b517-6f6ff64d89-qd7zj"
Nov 03 09:18:54.861707 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:54.861684    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"]
Nov 03 09:18:54.866664 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:54.866647    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"
Nov 03 09:18:54.915742 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:54.915713    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpjnw\" (UniqueName: \"kubernetes.io/projected/c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9-kube-api-access-lpjnw\") pod \"payload-processing-e2e-shared-a-w0-f9b517-6f6ff64d89-qd7zj\" (UID: \"c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f9b517-6f6ff64d89-qd7zj"
Nov 03 09:18:54.925091 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:54.925062    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"]
Nov 03 09:18:54.957349 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:54.957321    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7892f9a-7f14-4b08-b5a9-933572249e05-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp\" (UID: \"e7892f9a-7f14-4b08-b5a9-933572249e05\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"
Nov 03 09:18:54.957483 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:54.957436    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7892f9a-7f14-4b08-b5a9-933572249e05-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp\" (UID: \"e7892f9a-7f14-4b08-b5a9-933572249e05\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"
Nov 03 09:18:54.957483 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:54.957459    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6sdz\" (UniqueName: \"kubernetes.io/projected/e7892f9a-7f14-4b08-b5a9-933572249e05-kube-api-access-p6sdz\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp\" (UID: \"e7892f9a-7f14-4b08-b5a9-933572249e05\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"
Nov 03 09:18:54.957571 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:54.957488    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7892f9a-7f14-4b08-b5a9-933572249e05-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp\" (UID: \"e7892f9a-7f14-4b08-b5a9-933572249e05\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"
Nov 03 09:18:54.957627 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:54.957609    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7892f9a-7f14-4b08-b5a9-933572249e05-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp\" (UID: \"e7892f9a-7f14-4b08-b5a9-933572249e05\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"
Nov 03 09:18:54.957664 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:54.957642    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7892f9a-7f14-4b08-b5a9-933572249e05-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp\" (UID: \"e7892f9a-7f14-4b08-b5a9-933572249e05\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"
Nov 03 09:18:54.957698 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:54.957679    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7892f9a-7f14-4b08-b5a9-933572249e05-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp\" (UID: \"e7892f9a-7f14-4b08-b5a9-933572249e05\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"
Nov 03 09:18:54.957766 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:54.957755    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7892f9a-7f14-4b08-b5a9-933572249e05-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp\" (UID: \"e7892f9a-7f14-4b08-b5a9-933572249e05\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"
Nov 03 09:18:54.957798 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:54.957784    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7892f9a-7f14-4b08-b5a9-933572249e05-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp\" (UID: \"e7892f9a-7f14-4b08-b5a9-933572249e05\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"
Nov 03 09:18:54.988897 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:54.988820    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f9b517-6f6ff64d89-qd7zj"
Nov 03 09:18:55.058858 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:55.058830    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7892f9a-7f14-4b08-b5a9-933572249e05-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp\" (UID: \"e7892f9a-7f14-4b08-b5a9-933572249e05\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"
Nov 03 09:18:55.059038 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:55.058926    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7892f9a-7f14-4b08-b5a9-933572249e05-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp\" (UID: \"e7892f9a-7f14-4b08-b5a9-933572249e05\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"
Nov 03 09:18:55.059038 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:55.058971    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6sdz\" (UniqueName: \"kubernetes.io/projected/e7892f9a-7f14-4b08-b5a9-933572249e05-kube-api-access-p6sdz\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp\" (UID: \"e7892f9a-7f14-4b08-b5a9-933572249e05\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"
Nov 03 09:18:55.059038 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:55.059005    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7892f9a-7f14-4b08-b5a9-933572249e05-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp\" (UID: \"e7892f9a-7f14-4b08-b5a9-933572249e05\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"
Nov 03 09:18:55.059213 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:55.059041    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7892f9a-7f14-4b08-b5a9-933572249e05-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp\" (UID: \"e7892f9a-7f14-4b08-b5a9-933572249e05\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"
Nov 03 09:18:55.059213 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:55.059073    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7892f9a-7f14-4b08-b5a9-933572249e05-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp\" (UID: \"e7892f9a-7f14-4b08-b5a9-933572249e05\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"
Nov 03 09:18:55.059213 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:55.059130    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7892f9a-7f14-4b08-b5a9-933572249e05-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp\" (UID: \"e7892f9a-7f14-4b08-b5a9-933572249e05\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"
Nov 03 09:18:55.059370 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:55.059216    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7892f9a-7f14-4b08-b5a9-933572249e05-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp\" (UID: \"e7892f9a-7f14-4b08-b5a9-933572249e05\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"
Nov 03 09:18:55.059370 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:55.059263    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7892f9a-7f14-4b08-b5a9-933572249e05-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp\" (UID: \"e7892f9a-7f14-4b08-b5a9-933572249e05\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"
Nov 03 09:18:55.059370 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:55.059274    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7892f9a-7f14-4b08-b5a9-933572249e05-istio-data\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp\" (UID: \"e7892f9a-7f14-4b08-b5a9-933572249e05\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"
Nov 03 09:18:55.059549 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:55.059523    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7892f9a-7f14-4b08-b5a9-933572249e05-workload-certs\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp\" (UID: \"e7892f9a-7f14-4b08-b5a9-933572249e05\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"
Nov 03 09:18:55.060392 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:55.059932    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7892f9a-7f14-4b08-b5a9-933572249e05-workload-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp\" (UID: \"e7892f9a-7f14-4b08-b5a9-933572249e05\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"
Nov 03 09:18:55.060392 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:55.060154    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7892f9a-7f14-4b08-b5a9-933572249e05-credential-socket\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp\" (UID: \"e7892f9a-7f14-4b08-b5a9-933572249e05\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"
Nov 03 09:18:55.060392 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:55.060329    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7892f9a-7f14-4b08-b5a9-933572249e05-istiod-ca-cert\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp\" (UID: \"e7892f9a-7f14-4b08-b5a9-933572249e05\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"
Nov 03 09:18:55.062383 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:55.062342    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7892f9a-7f14-4b08-b5a9-933572249e05-istio-podinfo\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp\" (UID: \"e7892f9a-7f14-4b08-b5a9-933572249e05\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"
Nov 03 09:18:55.062868 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:55.062841    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7892f9a-7f14-4b08-b5a9-933572249e05-istio-envoy\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp\" (UID: \"e7892f9a-7f14-4b08-b5a9-933572249e05\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"
Nov 03 09:18:55.119705 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:55.119676    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7892f9a-7f14-4b08-b5a9-933572249e05-istio-token\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp\" (UID: \"e7892f9a-7f14-4b08-b5a9-933572249e05\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"
Nov 03 09:18:55.120302 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:55.120283    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6sdz\" (UniqueName: \"kubernetes.io/projected/e7892f9a-7f14-4b08-b5a9-933572249e05-kube-api-access-p6sdz\") pod \"e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp\" (UID: \"e7892f9a-7f14-4b08-b5a9-933572249e05\") " pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"
Nov 03 09:18:55.150673 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:55.150644    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f9b517-646c9dd495-qt8s8"]
Nov 03 09:18:55.151234 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:18:55.151208    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod842a3cc8_4062_41df_aa2f_18c19bb342c9.slice/crio-1b8666259751eb49f0e47c93be9b61dbcdfd6a4f3f3357c76526341b67ba4eec WatchSource:0}: Error finding container 1b8666259751eb49f0e47c93be9b61dbcdfd6a4f3f3357c76526341b67ba4eec: Status 404 returned error can't find the container with id 1b8666259751eb49f0e47c93be9b61dbcdfd6a4f3f3357c76526341b67ba4eec
Nov 03 09:18:55.188732 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:55.188707    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"
Nov 03 09:18:55.392741 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:55.392620    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-f9b517-6f6ff64d89-qd7zj"]
Nov 03 09:18:55.395245 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:18:55.395211    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc90b6d7d_56d5_4a0b_a4ac_295f1d36fce9.slice/crio-a703cfba713907cab1d77a55eb5d0d8b9a4cf0117354737c772de95c3f91ae95 WatchSource:0}: Error finding container a703cfba713907cab1d77a55eb5d0d8b9a4cf0117354737c772de95c3f91ae95: Status 404 returned error can't find the container with id a703cfba713907cab1d77a55eb5d0d8b9a4cf0117354737c772de95c3f91ae95
Nov 03 09:18:55.573283 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:55.573256    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"]
Nov 03 09:18:55.573607 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:18:55.573400    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7892f9a_7f14_4b08_b5a9_933572249e05.slice/crio-a587de4177b3ee6b4f594ca685849f92993407d2d5e83d9570a3e8783e9f5540 WatchSource:0}: Error finding container a587de4177b3ee6b4f594ca685849f92993407d2d5e83d9570a3e8783e9f5540: Status 404 returned error can't find the container with id a587de4177b3ee6b4f594ca685849f92993407d2d5e83d9570a3e8783e9f5540
Nov 03 09:18:55.575869 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:55.575835    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:18:55.575926 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:55.575893    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:18:55.575991 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:55.575927    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:18:55.707796 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:55.707755    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f9b517-646c9dd495-qt8s8" event={"ID":"842a3cc8-4062-41df-aa2f-18c19bb342c9","Type":"ContainerStarted","Data":"f2c9e77ac698d682bd952dc6b5c6ae98b7472e003a30d56b08871a6e62fbb0b9"}
Nov 03 09:18:55.708014 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:55.707821    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f9b517-646c9dd495-qt8s8" event={"ID":"842a3cc8-4062-41df-aa2f-18c19bb342c9","Type":"ContainerStarted","Data":"1b8666259751eb49f0e47c93be9b61dbcdfd6a4f3f3357c76526341b67ba4eec"}
Nov 03 09:18:55.708014 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:55.707984    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f9b517-646c9dd495-qt8s8"
Nov 03 09:18:55.709643 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:55.709613    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp" event={"ID":"e7892f9a-7f14-4b08-b5a9-933572249e05","Type":"ContainerStarted","Data":"fec4304477a974d327a33ea2db85a940312118b559b3131a0101301512a2b9c4"}
Nov 03 09:18:55.709741 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:55.709669    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp" event={"ID":"e7892f9a-7f14-4b08-b5a9-933572249e05","Type":"ContainerStarted","Data":"a587de4177b3ee6b4f594ca685849f92993407d2d5e83d9570a3e8783e9f5540"}
Nov 03 09:18:55.711363 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:55.711343    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f9b517-6f6ff64d89-qd7zj" event={"ID":"c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9","Type":"ContainerStarted","Data":"6b4eec6fcb2b235eb0db416abe970e09d2ef562e5ff2471f948db643aeb7a425"}
Nov 03 09:18:55.711470 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:55.711379    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f9b517-6f6ff64d89-qd7zj" event={"ID":"c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9","Type":"ContainerStarted","Data":"a703cfba713907cab1d77a55eb5d0d8b9a4cf0117354737c772de95c3f91ae95"}
Nov 03 09:18:55.711470 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:55.711409    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f9b517-6f6ff64d89-qd7zj"
Nov 03 09:18:55.823389 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:55.823276    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f9b517-646c9dd495-qt8s8" podStartSLOduration=1.823262625 podStartE2EDuration="1.823262625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:18:55.823013132 +0000 UTC m=+2360.092076569" watchObservedRunningTime="2026-11-03 09:18:55.823262625 +0000 UTC m=+2360.092326058"
Nov 03 09:18:55.970971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:55.962707    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp" podStartSLOduration=1.962683374 podStartE2EDuration="1.962683374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:18:55.959996532 +0000 UTC m=+2360.229059967" watchObservedRunningTime="2026-11-03 09:18:55.962683374 +0000 UTC m=+2360.231746811"
Nov 03 09:18:56.076343 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:56.076239    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f9b517-6f6ff64d89-qd7zj" podStartSLOduration=2.076221916 podStartE2EDuration="2.076221916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:18:56.075651575 +0000 UTC m=+2360.344715035" watchObservedRunningTime="2026-11-03 09:18:56.076221916 +0000 UTC m=+2360.345285352"
Nov 03 09:18:56.190389 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:56.190355    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"
Nov 03 09:18:56.198147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:56.198092    4473 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Nov 03 09:18:56.198425 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:56.198394    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp" podUID="e7892f9a-7f14-4b08-b5a9-933572249e05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Nov 03 09:18:57.189677 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:57.189641    4473 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Nov 03 09:18:57.190129 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:57.189694    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp" podUID="e7892f9a-7f14-4b08-b5a9-933572249e05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Nov 03 09:18:58.189441 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:58.189404    4473 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Nov 03 09:18:58.189641 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:58.189456    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp" podUID="e7892f9a-7f14-4b08-b5a9-933572249e05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Nov 03 09:18:59.189331 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:59.189296    4473 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Nov 03 09:18:59.189721 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:18:59.189348    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp" podUID="e7892f9a-7f14-4b08-b5a9-933572249e05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Nov 03 09:19:00.189647 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:00.189609    4473 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Nov 03 09:19:00.190087 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:00.189676    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp" podUID="e7892f9a-7f14-4b08-b5a9-933572249e05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Nov 03 09:19:01.189092 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:01.189058    4473 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Nov 03 09:19:01.189379 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:01.189115    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp" podUID="e7892f9a-7f14-4b08-b5a9-933572249e05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Nov 03 09:19:02.190075 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:02.190037    4473 patch_prober.go:29] interesting pod/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Nov 03 09:19:02.191279 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:02.191162    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp" podUID="e7892f9a-7f14-4b08-b5a9-933572249e05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Nov 03 09:19:03.193829 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:03.193802    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"
Nov 03 09:19:03.194253 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:03.194181    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"
Nov 03 09:19:03.194907 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:03.194887    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"
Nov 03 09:19:06.748834 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:06.748804    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f9b517-6f6ff64d89-qd7zj"
Nov 03 09:19:06.754198 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:06.754173    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f9b517-646c9dd495-qt8s8"
Nov 03 09:19:06.960551 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:06.960518    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:19:08.054799 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:08.054743    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b2ae518-9f65r"]
Nov 03 09:19:08.103700 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:08.103660    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b2ae518-9f65r"]
Nov 03 09:19:09.871839 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:09.871798    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d5440da-6d29-4e89-bfac-fd78fffed693" path="/var/lib/kubelet/pods/9d5440da-6d29-4e89-bfac-fd78fffed693/volumes"
Nov 03 09:19:16.937071 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:19:16.936992    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:19:27.055984 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:27.055931    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-107e604c-rn4sh"]
Nov 03 09:19:27.105827 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:27.105792    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-107e604c-rn4sh"]
Nov 03 09:19:27.870584 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:27.870554    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4a3260e-cf90-4fa4-883a-10894bb35df0" path="/var/lib/kubelet/pods/f4a3260e-cf90-4fa4-883a-10894bb35df0/volumes"
Nov 03 09:19:36.360317 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:36.360290    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 09:19:36.362202 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:36.362176    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 09:19:39.640877 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:39.640836    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"]
Nov 03 09:19:39.645933 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:39.645912    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"
Nov 03 09:19:39.674655 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:39.674625    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1224423e-4ddf-4902-80c7-19170efb4b38-workload-certs\") pod \"e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9\" (UID: \"1224423e-4ddf-4902-80c7-19170efb4b38\") " pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"
Nov 03 09:19:39.674837 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:39.674801    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1224423e-4ddf-4902-80c7-19170efb4b38-workload-socket\") pod \"e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9\" (UID: \"1224423e-4ddf-4902-80c7-19170efb4b38\") " pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"
Nov 03 09:19:39.674913 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:39.674870    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1224423e-4ddf-4902-80c7-19170efb4b38-istio-podinfo\") pod \"e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9\" (UID: \"1224423e-4ddf-4902-80c7-19170efb4b38\") " pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"
Nov 03 09:19:39.674983 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:39.674914    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1224423e-4ddf-4902-80c7-19170efb4b38-istiod-ca-cert\") pod \"e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9\" (UID: \"1224423e-4ddf-4902-80c7-19170efb4b38\") " pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"
Nov 03 09:19:39.675080 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:39.675060    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1224423e-4ddf-4902-80c7-19170efb4b38-istio-data\") pod \"e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9\" (UID: \"1224423e-4ddf-4902-80c7-19170efb4b38\") " pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"
Nov 03 09:19:39.675165 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:39.675123    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1224423e-4ddf-4902-80c7-19170efb4b38-istio-envoy\") pod \"e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9\" (UID: \"1224423e-4ddf-4902-80c7-19170efb4b38\") " pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"
Nov 03 09:19:39.675222 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:39.675166    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1224423e-4ddf-4902-80c7-19170efb4b38-istio-token\") pod \"e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9\" (UID: \"1224423e-4ddf-4902-80c7-19170efb4b38\") " pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"
Nov 03 09:19:39.675222 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:39.675212    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z6bz\" (UniqueName: \"kubernetes.io/projected/1224423e-4ddf-4902-80c7-19170efb4b38-kube-api-access-7z6bz\") pod \"e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9\" (UID: \"1224423e-4ddf-4902-80c7-19170efb4b38\") " pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"
Nov 03 09:19:39.675320 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:39.675250    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1224423e-4ddf-4902-80c7-19170efb4b38-credential-socket\") pod \"e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9\" (UID: \"1224423e-4ddf-4902-80c7-19170efb4b38\") " pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"
Nov 03 09:19:39.705287 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:39.705262    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-506e6d-openshift-default-dockercfg-8g6ks\""
Nov 03 09:19:39.761403 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:39.761371    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"]
Nov 03 09:19:39.775986 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:39.775925    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1224423e-4ddf-4902-80c7-19170efb4b38-istio-data\") pod \"e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9\" (UID: \"1224423e-4ddf-4902-80c7-19170efb4b38\") " pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"
Nov 03 09:19:39.776216 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:39.776191    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1224423e-4ddf-4902-80c7-19170efb4b38-istio-envoy\") pod \"e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9\" (UID: \"1224423e-4ddf-4902-80c7-19170efb4b38\") " pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"
Nov 03 09:19:39.776335 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:39.776234    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1224423e-4ddf-4902-80c7-19170efb4b38-istio-token\") pod \"e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9\" (UID: \"1224423e-4ddf-4902-80c7-19170efb4b38\") " pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"
Nov 03 09:19:39.776335 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:39.776269    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z6bz\" (UniqueName: \"kubernetes.io/projected/1224423e-4ddf-4902-80c7-19170efb4b38-kube-api-access-7z6bz\") pod \"e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9\" (UID: \"1224423e-4ddf-4902-80c7-19170efb4b38\") " pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"
Nov 03 09:19:39.776335 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:39.776292    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1224423e-4ddf-4902-80c7-19170efb4b38-credential-socket\") pod \"e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9\" (UID: \"1224423e-4ddf-4902-80c7-19170efb4b38\") " pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"
Nov 03 09:19:39.776515 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:39.776340    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1224423e-4ddf-4902-80c7-19170efb4b38-workload-certs\") pod \"e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9\" (UID: \"1224423e-4ddf-4902-80c7-19170efb4b38\") " pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"
Nov 03 09:19:39.776515 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:39.776388    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1224423e-4ddf-4902-80c7-19170efb4b38-istio-data\") pod \"e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9\" (UID: \"1224423e-4ddf-4902-80c7-19170efb4b38\") " pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"
Nov 03 09:19:39.776515 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:39.776455    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1224423e-4ddf-4902-80c7-19170efb4b38-workload-socket\") pod \"e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9\" (UID: \"1224423e-4ddf-4902-80c7-19170efb4b38\") " pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"
Nov 03 09:19:39.776682 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:39.776513    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1224423e-4ddf-4902-80c7-19170efb4b38-istio-podinfo\") pod \"e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9\" (UID: \"1224423e-4ddf-4902-80c7-19170efb4b38\") " pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"
Nov 03 09:19:39.776682 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:39.776557    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1224423e-4ddf-4902-80c7-19170efb4b38-istiod-ca-cert\") pod \"e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9\" (UID: \"1224423e-4ddf-4902-80c7-19170efb4b38\") " pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"
Nov 03 09:19:39.776803 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:39.776673    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1224423e-4ddf-4902-80c7-19170efb4b38-workload-certs\") pod \"e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9\" (UID: \"1224423e-4ddf-4902-80c7-19170efb4b38\") " pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"
Nov 03 09:19:39.776873 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:39.776851    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1224423e-4ddf-4902-80c7-19170efb4b38-credential-socket\") pod \"e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9\" (UID: \"1224423e-4ddf-4902-80c7-19170efb4b38\") " pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"
Nov 03 09:19:39.777132 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:39.776969    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1224423e-4ddf-4902-80c7-19170efb4b38-workload-socket\") pod \"e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9\" (UID: \"1224423e-4ddf-4902-80c7-19170efb4b38\") " pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"
Nov 03 09:19:39.777393 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:39.777355    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1224423e-4ddf-4902-80c7-19170efb4b38-istiod-ca-cert\") pod \"e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9\" (UID: \"1224423e-4ddf-4902-80c7-19170efb4b38\") " pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"
Nov 03 09:19:39.778789 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:39.778766    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1224423e-4ddf-4902-80c7-19170efb4b38-istio-envoy\") pod \"e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9\" (UID: \"1224423e-4ddf-4902-80c7-19170efb4b38\") " pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"
Nov 03 09:19:39.779424 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:39.779406    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1224423e-4ddf-4902-80c7-19170efb4b38-istio-podinfo\") pod \"e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9\" (UID: \"1224423e-4ddf-4902-80c7-19170efb4b38\") " pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"
Nov 03 09:19:39.900612 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:39.900385    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z6bz\" (UniqueName: \"kubernetes.io/projected/1224423e-4ddf-4902-80c7-19170efb4b38-kube-api-access-7z6bz\") pod \"e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9\" (UID: \"1224423e-4ddf-4902-80c7-19170efb4b38\") " pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"
Nov 03 09:19:39.934971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:39.934489    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1224423e-4ddf-4902-80c7-19170efb4b38-istio-token\") pod \"e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9\" (UID: \"1224423e-4ddf-4902-80c7-19170efb4b38\") " pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"
Nov 03 09:19:39.964970 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:39.962188    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"
Nov 03 09:19:40.489555 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:40.489522    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"]
Nov 03 09:19:40.490014 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:19:40.489988    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1224423e_4ddf_4902_80c7_19170efb4b38.slice/crio-cffb13f97c16b3cfdf7df6994d1bc56fc86326cc4f8742d062368b485a824cbb WatchSource:0}: Error finding container cffb13f97c16b3cfdf7df6994d1bc56fc86326cc4f8742d062368b485a824cbb: Status 404 returned error can't find the container with id cffb13f97c16b3cfdf7df6994d1bc56fc86326cc4f8742d062368b485a824cbb
Nov 03 09:19:40.492490 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:40.492456    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:19:40.492600 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:40.492524    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:19:40.492600 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:40.492558    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:19:41.104732 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:41.104684    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9" event={"ID":"1224423e-4ddf-4902-80c7-19170efb4b38","Type":"ContainerStarted","Data":"bf7662cc3fa8000a9e3178a6454d3618d2612e783aa508c9c80c1dd8b0a01f27"}
Nov 03 09:19:41.105157 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:41.104794    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9" event={"ID":"1224423e-4ddf-4902-80c7-19170efb4b38","Type":"ContainerStarted","Data":"cffb13f97c16b3cfdf7df6994d1bc56fc86326cc4f8742d062368b485a824cbb"}
Nov 03 09:19:41.227268 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:41.227218    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9" podStartSLOduration=2.227203959 podStartE2EDuration="2.227203959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:19:41.224163371 +0000 UTC m=+2405.493226828" watchObservedRunningTime="2026-11-03 09:19:41.227203959 +0000 UTC m=+2405.496267392"
Nov 03 09:19:41.963773 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:41.963747    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"
Nov 03 09:19:41.968913 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:41.968892    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"
Nov 03 09:19:42.111992 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:42.111961    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"
Nov 03 09:19:42.112698 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:42.112672    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"
Nov 03 09:19:45.053493 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:45.053460    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd5a7382-6z7sb"]
Nov 03 09:19:45.103329 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:45.103286    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd5a7382-6z7sb"]
Nov 03 09:19:45.870034 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:45.870005    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="447303f1-c173-4810-acfd-51993e615c0f" path="/var/lib/kubelet/pods/447303f1-c173-4810-acfd-51993e615c0f/volumes"
Nov 03 09:19:46.937179 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:19:46.937139    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:19:47.983824 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:47.983783    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-506e6d-767cbcb48c-dj25d"]
Nov 03 09:19:47.988684 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:47.988664    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-506e6d-767cbcb48c-dj25d"
Nov 03 09:19:48.044553 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:48.044525    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-506e6d-767cbcb48c-dj25d"]
Nov 03 09:19:48.097184 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:48.097128    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-506e6d\""
Nov 03 09:19:48.117982 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:48.101985    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5243dfa-09ba-4203-be20-66b1bdf296f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-506e6d-767cbcb48c-dj25d\" (UID: \"f5243dfa-09ba-4203-be20-66b1bdf296f8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-506e6d-767cbcb48c-dj25d"
Nov 03 09:19:48.117982 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:48.102041    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7jx6\" (UniqueName: \"kubernetes.io/projected/f5243dfa-09ba-4203-be20-66b1bdf296f8-kube-api-access-x7jx6\") pod \"payload-pre-processing-e2e-shared-b-w2-506e6d-767cbcb48c-dj25d\" (UID: \"f5243dfa-09ba-4203-be20-66b1bdf296f8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-506e6d-767cbcb48c-dj25d"
Nov 03 09:19:48.144648 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:48.144358    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-506e6d-dockercfg-hhff6\""
Nov 03 09:19:48.229591 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:48.217294    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5243dfa-09ba-4203-be20-66b1bdf296f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-506e6d-767cbcb48c-dj25d\" (UID: \"f5243dfa-09ba-4203-be20-66b1bdf296f8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-506e6d-767cbcb48c-dj25d"
Nov 03 09:19:48.229591 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:48.217348    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7jx6\" (UniqueName: \"kubernetes.io/projected/f5243dfa-09ba-4203-be20-66b1bdf296f8-kube-api-access-x7jx6\") pod \"payload-pre-processing-e2e-shared-b-w2-506e6d-767cbcb48c-dj25d\" (UID: \"f5243dfa-09ba-4203-be20-66b1bdf296f8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-506e6d-767cbcb48c-dj25d"
Nov 03 09:19:48.229591 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:48.218580    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5243dfa-09ba-4203-be20-66b1bdf296f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-506e6d-767cbcb48c-dj25d\" (UID: \"f5243dfa-09ba-4203-be20-66b1bdf296f8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-506e6d-767cbcb48c-dj25d"
Nov 03 09:19:48.229591 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:48.218631    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b64d08a9-h9ph8"]
Nov 03 09:19:48.230613 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:48.230539    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b64d08a9-h9ph8"]
Nov 03 09:19:48.260116 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:48.258160    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-506e6d-94df8cf47-69862"]
Nov 03 09:19:48.267634 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:48.263545    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-506e6d-94df8cf47-69862"
Nov 03 09:19:48.270855 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:48.270827    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-506e6d-94df8cf47-69862"]
Nov 03 09:19:48.277690 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:48.277665    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7jx6\" (UniqueName: \"kubernetes.io/projected/f5243dfa-09ba-4203-be20-66b1bdf296f8-kube-api-access-x7jx6\") pod \"payload-pre-processing-e2e-shared-b-w2-506e6d-767cbcb48c-dj25d\" (UID: \"f5243dfa-09ba-4203-be20-66b1bdf296f8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-506e6d-767cbcb48c-dj25d"
Nov 03 09:19:48.302539 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:48.302517    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-506e6d-767cbcb48c-dj25d"
Nov 03 09:19:48.318308 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:48.318285    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a1aba26-e2bf-43e8-b373-e3ab815bfd10-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-506e6d-94df8cf47-69862\" (UID: \"3a1aba26-e2bf-43e8-b373-e3ab815bfd10\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-506e6d-94df8cf47-69862"
Nov 03 09:19:48.318554 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:48.318535    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdm2h\" (UniqueName: \"kubernetes.io/projected/3a1aba26-e2bf-43e8-b373-e3ab815bfd10-kube-api-access-vdm2h\") pod \"payload-processing-e2e-shared-b-w2-506e6d-94df8cf47-69862\" (UID: \"3a1aba26-e2bf-43e8-b373-e3ab815bfd10\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-506e6d-94df8cf47-69862"
Nov 03 09:19:48.420036 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:48.420002    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdm2h\" (UniqueName: \"kubernetes.io/projected/3a1aba26-e2bf-43e8-b373-e3ab815bfd10-kube-api-access-vdm2h\") pod \"payload-processing-e2e-shared-b-w2-506e6d-94df8cf47-69862\" (UID: \"3a1aba26-e2bf-43e8-b373-e3ab815bfd10\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-506e6d-94df8cf47-69862"
Nov 03 09:19:48.420235 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:48.420053    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a1aba26-e2bf-43e8-b373-e3ab815bfd10-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-506e6d-94df8cf47-69862\" (UID: \"3a1aba26-e2bf-43e8-b373-e3ab815bfd10\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-506e6d-94df8cf47-69862"
Nov 03 09:19:48.421003 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:48.420980    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a1aba26-e2bf-43e8-b373-e3ab815bfd10-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-506e6d-94df8cf47-69862\" (UID: \"3a1aba26-e2bf-43e8-b373-e3ab815bfd10\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-506e6d-94df8cf47-69862"
Nov 03 09:19:48.479240 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:48.479207    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdm2h\" (UniqueName: \"kubernetes.io/projected/3a1aba26-e2bf-43e8-b373-e3ab815bfd10-kube-api-access-vdm2h\") pod \"payload-processing-e2e-shared-b-w2-506e6d-94df8cf47-69862\" (UID: \"3a1aba26-e2bf-43e8-b373-e3ab815bfd10\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-506e6d-94df8cf47-69862"
Nov 03 09:19:48.580807 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:48.580736    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-506e6d-94df8cf47-69862"
Nov 03 09:19:48.681666 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:48.680208    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-506e6d-767cbcb48c-dj25d"]
Nov 03 09:19:48.683650 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:19:48.683621    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5243dfa_09ba_4203_be20_66b1bdf296f8.slice/crio-907f94751125b72fb444fb57884c4bd7b9e19c1512b36162aef4042792bdbab9 WatchSource:0}: Error finding container 907f94751125b72fb444fb57884c4bd7b9e19c1512b36162aef4042792bdbab9: Status 404 returned error can't find the container with id 907f94751125b72fb444fb57884c4bd7b9e19c1512b36162aef4042792bdbab9
Nov 03 09:19:48.958958 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:19:48.958920    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a1aba26_e2bf_43e8_b373_e3ab815bfd10.slice/crio-b9012cdd5487b86edbf5ac63ec290aafeef62d977b86322bdbe91586593d38db WatchSource:0}: Error finding container b9012cdd5487b86edbf5ac63ec290aafeef62d977b86322bdbe91586593d38db: Status 404 returned error can't find the container with id b9012cdd5487b86edbf5ac63ec290aafeef62d977b86322bdbe91586593d38db
Nov 03 09:19:48.959276 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:48.959255    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-506e6d-94df8cf47-69862"]
Nov 03 09:19:49.229747 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:49.229650    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-506e6d-767cbcb48c-dj25d" event={"ID":"f5243dfa-09ba-4203-be20-66b1bdf296f8","Type":"ContainerStarted","Data":"27c5518da647adb7fc3e6510fbb4a2e9f74a6a49b9784291b3077cd28b3f3ea1"}
Nov 03 09:19:49.229747 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:49.229710    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-506e6d-767cbcb48c-dj25d" event={"ID":"f5243dfa-09ba-4203-be20-66b1bdf296f8","Type":"ContainerStarted","Data":"907f94751125b72fb444fb57884c4bd7b9e19c1512b36162aef4042792bdbab9"}
Nov 03 09:19:49.230297 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:49.229849    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-506e6d-767cbcb48c-dj25d"
Nov 03 09:19:49.231753 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:49.231731    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-506e6d-94df8cf47-69862" event={"ID":"3a1aba26-e2bf-43e8-b373-e3ab815bfd10","Type":"ContainerStarted","Data":"9b8e7279b28ecf8f62deca2c3d0473591b884bb73325e0eff9c930f96a08a343"}
Nov 03 09:19:49.231860 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:49.231768    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-506e6d-94df8cf47-69862" event={"ID":"3a1aba26-e2bf-43e8-b373-e3ab815bfd10","Type":"ContainerStarted","Data":"b9012cdd5487b86edbf5ac63ec290aafeef62d977b86322bdbe91586593d38db"}
Nov 03 09:19:49.231860 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:49.231798    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-506e6d-94df8cf47-69862"
Nov 03 09:19:49.347514 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:49.347275    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-506e6d-767cbcb48c-dj25d" podStartSLOduration=2.347256088 podStartE2EDuration="2.347256088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:19:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:19:49.34549646 +0000 UTC m=+2413.614559894" watchObservedRunningTime="2026-11-03 09:19:49.347256088 +0000 UTC m=+2413.616319523"
Nov 03 09:19:49.870788 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:49.870753    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a936a741-17a8-45d5-9828-4891c861cc88" path="/var/lib/kubelet/pods/a936a741-17a8-45d5-9828-4891c861cc88/volumes"
Nov 03 09:19:52.414181 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:52.414127    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-506e6d-94df8cf47-69862" podStartSLOduration=4.414110386 podStartE2EDuration="4.414110386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:19:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:19:49.478271801 +0000 UTC m=+2413.747335283" watchObservedRunningTime="2026-11-03 09:19:52.414110386 +0000 UTC m=+2416.683173821"
Nov 03 09:19:52.417699 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:19:52.417676    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:20:00.241558 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:00.241529    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-506e6d-767cbcb48c-dj25d"
Nov 03 09:20:00.241980 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:00.241601    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-506e6d-94df8cf47-69862"
Nov 03 09:20:02.075085 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:02.075047    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5de7da4-r4976"]
Nov 03 09:20:02.124860 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:02.124823    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5de7da4-r4976"]
Nov 03 09:20:03.871781 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:03.871746    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2" path="/var/lib/kubelet/pods/31d9edb0-a5e7-4f7d-9fac-3e1c1f62c5c2/volumes"
Nov 03 09:20:13.180903 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:13.180864    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"]
Nov 03 09:20:13.185674 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:13.185634    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"
Nov 03 09:20:13.246340 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:13.246308    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-3df1cc-openshift-default-dockercfg-55jhg\""
Nov 03 09:20:13.319968 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:13.307341    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-workload-socket\") pod \"e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v\" (UID: \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\") " pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"
Nov 03 09:20:13.319968 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:13.307407    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-istio-podinfo\") pod \"e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v\" (UID: \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\") " pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"
Nov 03 09:20:13.319968 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:13.307494    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-istio-token\") pod \"e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v\" (UID: \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\") " pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"
Nov 03 09:20:13.319968 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:13.307564    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-istio-data\") pod \"e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v\" (UID: \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\") " pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"
Nov 03 09:20:13.319968 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:13.307648    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-istio-envoy\") pod \"e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v\" (UID: \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\") " pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"
Nov 03 09:20:13.319968 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:13.307678    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-istiod-ca-cert\") pod \"e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v\" (UID: \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\") " pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"
Nov 03 09:20:13.319968 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:13.307723    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf4p8\" (UniqueName: \"kubernetes.io/projected/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-kube-api-access-cf4p8\") pod \"e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v\" (UID: \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\") " pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"
Nov 03 09:20:13.319968 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:13.307837    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-credential-socket\") pod \"e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v\" (UID: \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\") " pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"
Nov 03 09:20:13.319968 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:13.307909    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-workload-certs\") pod \"e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v\" (UID: \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\") " pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"
Nov 03 09:20:13.382500 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:13.382464    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"]
Nov 03 09:20:13.416190 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:13.411843    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-istio-envoy\") pod \"e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v\" (UID: \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\") " pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"
Nov 03 09:20:13.416564 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:13.416506    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-istiod-ca-cert\") pod \"e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v\" (UID: \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\") " pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"
Nov 03 09:20:13.416751 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:13.416732    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf4p8\" (UniqueName: \"kubernetes.io/projected/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-kube-api-access-cf4p8\") pod \"e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v\" (UID: \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\") " pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"
Nov 03 09:20:13.416959 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:13.416926    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-credential-socket\") pod \"e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v\" (UID: \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\") " pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"
Nov 03 09:20:13.417117 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:13.417102    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-workload-certs\") pod \"e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v\" (UID: \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\") " pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"
Nov 03 09:20:13.417360 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:13.417340    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-workload-socket\") pod \"e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v\" (UID: \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\") " pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"
Nov 03 09:20:13.417497 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:13.417483    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-istio-podinfo\") pod \"e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v\" (UID: \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\") " pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"
Nov 03 09:20:13.417647 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:13.417632    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-istio-token\") pod \"e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v\" (UID: \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\") " pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"
Nov 03 09:20:13.417777 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:13.417763    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-istio-data\") pod \"e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v\" (UID: \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\") " pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"
Nov 03 09:20:13.471088 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:13.469401    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-istio-data\") pod \"e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v\" (UID: \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\") " pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"
Nov 03 09:20:13.493020 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:13.471891    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-istiod-ca-cert\") pod \"e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v\" (UID: \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\") " pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"
Nov 03 09:20:13.497266 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:13.493589    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-workload-certs\") pod \"e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v\" (UID: \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\") " pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"
Nov 03 09:20:13.497516 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:13.494692    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-credential-socket\") pod \"e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v\" (UID: \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\") " pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"
Nov 03 09:20:13.498480 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:13.498457    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-workload-socket\") pod \"e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v\" (UID: \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\") " pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"
Nov 03 09:20:13.510194 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:13.510096    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-istio-podinfo\") pod \"e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v\" (UID: \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\") " pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"
Nov 03 09:20:13.517913 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:13.517855    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf4p8\" (UniqueName: \"kubernetes.io/projected/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-kube-api-access-cf4p8\") pod \"e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v\" (UID: \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\") " pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"
Nov 03 09:20:13.549906 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:13.549833    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-istio-envoy\") pod \"e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v\" (UID: \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\") " pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"
Nov 03 09:20:13.552120 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:13.552086    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-istio-token\") pod \"e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v\" (UID: \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\") " pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"
Nov 03 09:20:13.570918 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:13.570891    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"
Nov 03 09:20:14.103539 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:14.103495    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"]
Nov 03 09:20:14.119044 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:14.118996    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:20:14.119185 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:14.119088    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:20:14.119185 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:14.119133    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:20:14.560605 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:14.560565    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v" event={"ID":"603f6f44-b6ae-4d49-b79d-e73bb6d853a9","Type":"ContainerStarted","Data":"6651d9c20d2ae8db25f249ae0b3c4ed341bc86f976668be7c11ec8b1183a53d1"}
Nov 03 09:20:14.561065 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:14.560633    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v" event={"ID":"603f6f44-b6ae-4d49-b79d-e73bb6d853a9","Type":"ContainerStarted","Data":"46deb179715cbf91fc7f043e4ba26a6e0a52e0b5be0b95c664abd2833a5125d6"}
Nov 03 09:20:14.572013 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:14.571991    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"
Nov 03 09:20:14.682200 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:14.682139    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v" podStartSLOduration=1.682124113 podStartE2EDuration="1.682124113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:20:14.679932153 +0000 UTC m=+2438.948995644" watchObservedRunningTime="2026-11-03 09:20:14.682124113 +0000 UTC m=+2438.951187547"
Nov 03 09:20:15.577145 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:15.577119    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"
Nov 03 09:20:16.573495 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:16.573468    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"
Nov 03 09:20:16.574456 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:16.574438    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"
Nov 03 09:20:16.938459 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:20:16.938426    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:20:18.054651 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:18.054619    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c826a7d-47qmk"]
Nov 03 09:20:18.104538 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:18.104506    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c826a7d-47qmk"]
Nov 03 09:20:19.871317 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:19.871284    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17acd40d-9a11-4a0e-9a09-88f9756e2402" path="/var/lib/kubelet/pods/17acd40d-9a11-4a0e-9a09-88f9756e2402/volumes"
Nov 03 09:20:26.495430 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:26.494365    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-3df1cc-6d58d67765-44gqp"]
Nov 03 09:20:26.548455 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:26.538965    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-3df1cc-6d58d67765-44gqp"]
Nov 03 09:20:26.548455 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:26.539097    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3df1cc-6d58d67765-44gqp"
Nov 03 09:20:26.643194 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:26.643155    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-3df1cc\""
Nov 03 09:20:26.643194 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:26.643164    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-3df1cc-dockercfg-pkqg5\""
Nov 03 09:20:26.654898 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:26.654863    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdf54\" (UniqueName: \"kubernetes.io/projected/9f2cb2de-df5b-4f79-95f1-4232463e82bb-kube-api-access-zdf54\") pod \"payload-processing-e2e-shared-b-w1-3df1cc-6d58d67765-44gqp\" (UID: \"9f2cb2de-df5b-4f79-95f1-4232463e82bb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3df1cc-6d58d67765-44gqp"
Nov 03 09:20:26.655123 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:26.655100    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f2cb2de-df5b-4f79-95f1-4232463e82bb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-3df1cc-6d58d67765-44gqp\" (UID: \"9f2cb2de-df5b-4f79-95f1-4232463e82bb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3df1cc-6d58d67765-44gqp"
Nov 03 09:20:26.756629 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:26.756535    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f2cb2de-df5b-4f79-95f1-4232463e82bb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-3df1cc-6d58d67765-44gqp\" (UID: \"9f2cb2de-df5b-4f79-95f1-4232463e82bb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3df1cc-6d58d67765-44gqp"
Nov 03 09:20:26.756822 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:26.756704    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdf54\" (UniqueName: \"kubernetes.io/projected/9f2cb2de-df5b-4f79-95f1-4232463e82bb-kube-api-access-zdf54\") pod \"payload-processing-e2e-shared-b-w1-3df1cc-6d58d67765-44gqp\" (UID: \"9f2cb2de-df5b-4f79-95f1-4232463e82bb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3df1cc-6d58d67765-44gqp"
Nov 03 09:20:26.757467 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:26.757447    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f2cb2de-df5b-4f79-95f1-4232463e82bb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-3df1cc-6d58d67765-44gqp\" (UID: \"9f2cb2de-df5b-4f79-95f1-4232463e82bb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3df1cc-6d58d67765-44gqp"
Nov 03 09:20:26.816147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:26.816110    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdf54\" (UniqueName: \"kubernetes.io/projected/9f2cb2de-df5b-4f79-95f1-4232463e82bb-kube-api-access-zdf54\") pod \"payload-processing-e2e-shared-b-w1-3df1cc-6d58d67765-44gqp\" (UID: \"9f2cb2de-df5b-4f79-95f1-4232463e82bb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3df1cc-6d58d67765-44gqp"
Nov 03 09:20:26.871261 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:26.871220    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3df1cc-6d58d67765-44gqp"
Nov 03 09:20:27.257479 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:20:27.257441    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f2cb2de_df5b_4f79_95f1_4232463e82bb.slice/crio-b7eda0b52c99f21fd663ad57e37bdc264cbedcb5d30f956211774a7ad609688d WatchSource:0}: Error finding container b7eda0b52c99f21fd663ad57e37bdc264cbedcb5d30f956211774a7ad609688d: Status 404 returned error can't find the container with id b7eda0b52c99f21fd663ad57e37bdc264cbedcb5d30f956211774a7ad609688d
Nov 03 09:20:27.258478 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:27.258439    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-3df1cc-6d58d67765-44gqp"]
Nov 03 09:20:27.675679 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:27.675627    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3df1cc-6d58d67765-44gqp" event={"ID":"9f2cb2de-df5b-4f79-95f1-4232463e82bb","Type":"ContainerStarted","Data":"9b04779f0939659e793305130e31d8046b84cd548035b00bb776d99cb6770ec1"}
Nov 03 09:20:27.676208 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:27.675740    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3df1cc-6d58d67765-44gqp" event={"ID":"9f2cb2de-df5b-4f79-95f1-4232463e82bb","Type":"ContainerStarted","Data":"b7eda0b52c99f21fd663ad57e37bdc264cbedcb5d30f956211774a7ad609688d"}
Nov 03 09:20:27.676208 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:27.676197    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3df1cc-6d58d67765-44gqp"
Nov 03 09:20:27.795150 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:27.795086    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3df1cc-6d58d67765-44gqp" podStartSLOduration=1.795071906 podStartE2EDuration="1.795071906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:20:27.791639521 +0000 UTC m=+2452.060702956" watchObservedRunningTime="2026-11-03 09:20:27.795071906 +0000 UTC m=+2452.064135389"
Nov 03 09:20:28.082492 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:28.082397    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b2d37a3-zglmc"]
Nov 03 09:20:28.089297 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:28.089263    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b2d37a3-zglmc"]
Nov 03 09:20:29.533098 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:29.533030    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"]
Nov 03 09:20:29.533574 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:29.533241    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2" podUID="94998825-9ad4-4f71-9c54-b4c2a5032c26" containerName="istio-proxy" containerID="cri-o://0b56a3c4a3f67bd48d72134dcfb0a424cc3a3f209541161ad7650c737001076a" gracePeriod=30
Nov 03 09:20:29.689762 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:29.689730    4473 generic.go:356] "Generic (PLEG): container finished" podID="909a4fb7-462b-4d1b-a96f-8da66cb52d11" containerID="0ecc964583511d973a3acbc4c4e2030887d6841e0d68f884b85cef3bb1e5c8ce" exitCode=0
Nov 03 09:20:29.690002 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:29.689815    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd" event={"ID":"909a4fb7-462b-4d1b-a96f-8da66cb52d11","Type":"ContainerDied","Data":"0ecc964583511d973a3acbc4c4e2030887d6841e0d68f884b85cef3bb1e5c8ce"}
Nov 03 09:20:29.690274 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:29.690260    4473 scope.go:122] "RemoveContainer" containerID="0ecc964583511d973a3acbc4c4e2030887d6841e0d68f884b85cef3bb1e5c8ce"
Nov 03 09:20:29.691861 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:29.691830    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:20:29.692035 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:29.691910    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:20:29.692035 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:29.691997    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:20:29.882366 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:29.882331    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0103659f-3024-4869-83f5-4fed041510bc" path="/var/lib/kubelet/pods/0103659f-3024-4869-83f5-4fed041510bc/volumes"
Nov 03 09:20:30.699399 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:30.699366    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd" event={"ID":"909a4fb7-462b-4d1b-a96f-8da66cb52d11","Type":"ContainerStarted","Data":"a65d2b3382a6f7ce75330c2afcb855315ef5bcbbb37dc8f4976b13a075acab53"}
Nov 03 09:20:31.065866 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:31.065780    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"
Nov 03 09:20:31.067113 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:31.067093    4473 patch_prober.go:29] interesting pod/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 09:20:31.067192 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:31.067138    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd" podUID="909a4fb7-462b-4d1b-a96f-8da66cb52d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 09:20:32.065937 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:32.065896    4473 patch_prober.go:29] interesting pod/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 09:20:32.066371 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:32.065970    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd" podUID="909a4fb7-462b-4d1b-a96f-8da66cb52d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 09:20:32.653973 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:32.653078    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"
Nov 03 09:20:33.066256 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:33.066166    4473 patch_prober.go:29] interesting pod/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 09:20:33.066256 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:33.066228    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd" podUID="909a4fb7-462b-4d1b-a96f-8da66cb52d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 09:20:34.065629 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:34.065595    4473 patch_prober.go:29] interesting pod/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 09:20:34.065822 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:34.065659    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd" podUID="909a4fb7-462b-4d1b-a96f-8da66cb52d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 09:20:34.283217 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:34.283183    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:20:34.587339 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:20:34.587304    4473 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94998825_9ad4_4f71_9c54_b4c2a5032c26.slice/crio-conmon-0b56a3c4a3f67bd48d72134dcfb0a424cc3a3f209541161ad7650c737001076a.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 09:20:34.587339 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:20:34.587325    4473 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94998825_9ad4_4f71_9c54_b4c2a5032c26.slice/crio-conmon-0b56a3c4a3f67bd48d72134dcfb0a424cc3a3f209541161ad7650c737001076a.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 09:20:34.756272 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:34.754891    4473 generic.go:356] "Generic (PLEG): container finished" podID="94998825-9ad4-4f71-9c54-b4c2a5032c26" containerID="0b56a3c4a3f67bd48d72134dcfb0a424cc3a3f209541161ad7650c737001076a" exitCode=0
Nov 03 09:20:34.756272 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:34.755069    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2" event={"ID":"94998825-9ad4-4f71-9c54-b4c2a5032c26","Type":"ContainerDied","Data":"0b56a3c4a3f67bd48d72134dcfb0a424cc3a3f209541161ad7650c737001076a"}
Nov 03 09:20:34.850432 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:34.850401    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"
Nov 03 09:20:34.896377 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:34.896350    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94998825-9ad4-4f71-9c54-b4c2a5032c26-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94998825-9ad4-4f71-9c54-b4c2a5032c26-istio-data\") pod \"94998825-9ad4-4f71-9c54-b4c2a5032c26\" (UID: \"94998825-9ad4-4f71-9c54-b4c2a5032c26\") "
Nov 03 09:20:34.896546 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:34.896405    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94998825-9ad4-4f71-9c54-b4c2a5032c26-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94998825-9ad4-4f71-9c54-b4c2a5032c26-istiod-ca-cert\") pod \"94998825-9ad4-4f71-9c54-b4c2a5032c26\" (UID: \"94998825-9ad4-4f71-9c54-b4c2a5032c26\") "
Nov 03 09:20:34.896546 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:34.896482    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94998825-9ad4-4f71-9c54-b4c2a5032c26-kube-api-access-jjjtm\" (UniqueName: \"kubernetes.io/projected/94998825-9ad4-4f71-9c54-b4c2a5032c26-kube-api-access-jjjtm\") pod \"94998825-9ad4-4f71-9c54-b4c2a5032c26\" (UID: \"94998825-9ad4-4f71-9c54-b4c2a5032c26\") "
Nov 03 09:20:34.896546 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:34.896523    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94998825-9ad4-4f71-9c54-b4c2a5032c26-istio-token\" (UniqueName: \"kubernetes.io/projected/94998825-9ad4-4f71-9c54-b4c2a5032c26-istio-token\") pod \"94998825-9ad4-4f71-9c54-b4c2a5032c26\" (UID: \"94998825-9ad4-4f71-9c54-b4c2a5032c26\") "
Nov 03 09:20:34.896697 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:34.896589    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94998825-9ad4-4f71-9c54-b4c2a5032c26-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94998825-9ad4-4f71-9c54-b4c2a5032c26-istio-envoy\") pod \"94998825-9ad4-4f71-9c54-b4c2a5032c26\" (UID: \"94998825-9ad4-4f71-9c54-b4c2a5032c26\") "
Nov 03 09:20:34.896697 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:34.896610    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94998825-9ad4-4f71-9c54-b4c2a5032c26-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94998825-9ad4-4f71-9c54-b4c2a5032c26-credential-socket\") pod \"94998825-9ad4-4f71-9c54-b4c2a5032c26\" (UID: \"94998825-9ad4-4f71-9c54-b4c2a5032c26\") "
Nov 03 09:20:34.896697 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:34.896626    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94998825-9ad4-4f71-9c54-b4c2a5032c26-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94998825-9ad4-4f71-9c54-b4c2a5032c26-workload-socket\") pod \"94998825-9ad4-4f71-9c54-b4c2a5032c26\" (UID: \"94998825-9ad4-4f71-9c54-b4c2a5032c26\") "
Nov 03 09:20:34.896697 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:34.896652    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94998825-9ad4-4f71-9c54-b4c2a5032c26-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94998825-9ad4-4f71-9c54-b4c2a5032c26-istio-podinfo\") pod \"94998825-9ad4-4f71-9c54-b4c2a5032c26\" (UID: \"94998825-9ad4-4f71-9c54-b4c2a5032c26\") "
Nov 03 09:20:34.896697 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:34.896657    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94998825-9ad4-4f71-9c54-b4c2a5032c26-istio-data" pod "94998825-9ad4-4f71-9c54-b4c2a5032c26" (UID: "94998825-9ad4-4f71-9c54-b4c2a5032c26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:20:34.896697 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:34.896677    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94998825-9ad4-4f71-9c54-b4c2a5032c26-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94998825-9ad4-4f71-9c54-b4c2a5032c26-workload-certs\") pod \"94998825-9ad4-4f71-9c54-b4c2a5032c26\" (UID: \"94998825-9ad4-4f71-9c54-b4c2a5032c26\") "
Nov 03 09:20:34.897040 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:34.896906    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94998825-9ad4-4f71-9c54-b4c2a5032c26-istiod-ca-cert" pod "94998825-9ad4-4f71-9c54-b4c2a5032c26" (UID: "94998825-9ad4-4f71-9c54-b4c2a5032c26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:20:34.897100 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:34.897075    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94998825-9ad4-4f71-9c54-b4c2a5032c26-workload-socket" pod "94998825-9ad4-4f71-9c54-b4c2a5032c26" (UID: "94998825-9ad4-4f71-9c54-b4c2a5032c26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:20:34.897290 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:34.897263    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94998825-9ad4-4f71-9c54-b4c2a5032c26-workload-certs" pod "94998825-9ad4-4f71-9c54-b4c2a5032c26" (UID: "94998825-9ad4-4f71-9c54-b4c2a5032c26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:20:34.897425 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:34.897403    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94998825-9ad4-4f71-9c54-b4c2a5032c26-credential-socket" pod "94998825-9ad4-4f71-9c54-b4c2a5032c26" (UID: "94998825-9ad4-4f71-9c54-b4c2a5032c26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:20:34.897516 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:34.897497    4473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94998825-9ad4-4f71-9c54-b4c2a5032c26-workload-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:20:34.897516 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:34.897513    4473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94998825-9ad4-4f71-9c54-b4c2a5032c26-workload-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:20:34.897641 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:34.897523    4473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94998825-9ad4-4f71-9c54-b4c2a5032c26-istio-data\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:20:34.897641 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:34.897532    4473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94998825-9ad4-4f71-9c54-b4c2a5032c26-istiod-ca-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:20:34.899351 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:34.899323    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94998825-9ad4-4f71-9c54-b4c2a5032c26-istio-envoy" pod "94998825-9ad4-4f71-9c54-b4c2a5032c26" (UID: "94998825-9ad4-4f71-9c54-b4c2a5032c26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:20:34.899585 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:34.899564    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94998825-9ad4-4f71-9c54-b4c2a5032c26-istio-podinfo" pod "94998825-9ad4-4f71-9c54-b4c2a5032c26" (UID: "94998825-9ad4-4f71-9c54-b4c2a5032c26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 09:20:34.899644 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:34.899594    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94998825-9ad4-4f71-9c54-b4c2a5032c26-istio-token" pod "94998825-9ad4-4f71-9c54-b4c2a5032c26" (UID: "94998825-9ad4-4f71-9c54-b4c2a5032c26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:20:34.900304 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:34.900287    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94998825-9ad4-4f71-9c54-b4c2a5032c26-kube-api-access-jjjtm" pod "94998825-9ad4-4f71-9c54-b4c2a5032c26" (UID: "94998825-9ad4-4f71-9c54-b4c2a5032c26"). InnerVolumeSpecName "kube-api-access-jjjtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:20:34.998965 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:34.998908    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjjtm\" (UniqueName: \"kubernetes.io/projected/94998825-9ad4-4f71-9c54-b4c2a5032c26-kube-api-access-jjjtm\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:20:34.998965 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:34.998934    4473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94998825-9ad4-4f71-9c54-b4c2a5032c26-istio-token\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:20:34.999151 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:34.998992    4473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94998825-9ad4-4f71-9c54-b4c2a5032c26-istio-envoy\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:20:34.999151 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:34.999007    4473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94998825-9ad4-4f71-9c54-b4c2a5032c26-credential-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:20:34.999151 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:34.999019    4473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94998825-9ad4-4f71-9c54-b4c2a5032c26-istio-podinfo\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:20:35.065647 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:35.065561    4473 patch_prober.go:29] interesting pod/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 09:20:35.065647 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:35.065612    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd" podUID="909a4fb7-462b-4d1b-a96f-8da66cb52d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 09:20:35.762416 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:35.762386    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"
Nov 03 09:20:35.762913 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:35.762428    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2" event={"ID":"94998825-9ad4-4f71-9c54-b4c2a5032c26","Type":"ContainerDied","Data":"e4b7fb1df61d224a6f3653a276f7c97e61eb01bc7458ef3de4021397c617547e"}
Nov 03 09:20:35.762913 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:35.762495    4473 scope.go:122] "RemoveContainer" containerID="0b56a3c4a3f67bd48d72134dcfb0a424cc3a3f209541161ad7650c737001076a"
Nov 03 09:20:35.961584 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:35.961552    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"]
Nov 03 09:20:36.010107 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:36.010076    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c6bh2"]
Nov 03 09:20:36.066406 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:36.066320    4473 patch_prober.go:29] interesting pod/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 09:20:36.066406 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:36.066390    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd" podUID="909a4fb7-462b-4d1b-a96f-8da66cb52d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 09:20:37.071139 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:37.071111    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"
Nov 03 09:20:37.072169 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:37.072149    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"
Nov 03 09:20:37.612417 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:37.612390    4473 scope.go:122] "RemoveContainer" containerID="0058eb1144a3fd2dbfb6a2445140d306cf409fd675fc83bf2c44d710c93f139e"
Nov 03 09:20:37.621930 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:37.621911    4473 scope.go:122] "RemoveContainer" containerID="c9bc9a74c085b680eecc2729927b9d536ace672daf2c2d48b79d618410005c54"
Nov 03 09:20:37.869665 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:37.869639    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94998825-9ad4-4f71-9c54-b4c2a5032c26" path="/var/lib/kubelet/pods/94998825-9ad4-4f71-9c54-b4c2a5032c26/volumes"
Nov 03 09:20:38.685349 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:38.685321    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3df1cc-6d58d67765-44gqp"
Nov 03 09:20:45.054518 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:45.054470    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d038cef-bm85g"]
Nov 03 09:20:45.104782 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:45.104755    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d038cef-bm85g"]
Nov 03 09:20:45.901499 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:45.901464    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42f023db-6a98-4a5f-84d3-c9b30fc68658" path="/var/lib/kubelet/pods/42f023db-6a98-4a5f-84d3-c9b30fc68658/volumes"
Nov 03 09:20:46.938179 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:20:46.938106    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:20:49.052815 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:49.052776    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7d51490-2kvg2"]
Nov 03 09:20:49.104861 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:49.104835    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7d51490-2kvg2"]
Nov 03 09:20:49.870559 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:20:49.870514    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a5eb55a-8ab5-4d98-b409-87a08427dede" path="/var/lib/kubelet/pods/5a5eb55a-8ab5-4d98-b409-87a08427dede/volumes"
Nov 03 09:21:07.057103 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:07.057055    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f38978da-fmzch"]
Nov 03 09:21:07.105727 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:07.105695    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f38978da-fmzch"]
Nov 03 09:21:07.870248 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:07.870216    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="107cfccf-d2bf-4b2d-a64d-655b12b33fa8" path="/var/lib/kubelet/pods/107cfccf-d2bf-4b2d-a64d-655b12b33fa8/volumes"
Nov 03 09:21:12.052234 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:12.052201    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0591542-ggbkb"]
Nov 03 09:21:12.102305 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:12.102273    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0591542-ggbkb"]
Nov 03 09:21:13.870459 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:13.870426    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f8b6605-ed6f-494d-968f-bbd776b7057d" path="/var/lib/kubelet/pods/4f8b6605-ed6f-494d-968f-bbd776b7057d/volumes"
Nov 03 09:21:16.940185 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:21:16.940148    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:21:21.339970 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:21.335306    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:21:24.095148 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:24.095111    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01640f00-7pfvq"]
Nov 03 09:21:24.098116 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:24.098092    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01640f00-7pfvq"]
Nov 03 09:21:25.923968 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:25.923744    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="212f6063-6386-4d83-9da1-3030eb8c1482" path="/var/lib/kubelet/pods/212f6063-6386-4d83-9da1-3030eb8c1482/volumes"
Nov 03 09:21:32.452069 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:32.452036    4473 generic.go:356] "Generic (PLEG): container finished" podID="1224423e-4ddf-4902-80c7-19170efb4b38" containerID="bf7662cc3fa8000a9e3178a6454d3618d2612e783aa508c9c80c1dd8b0a01f27" exitCode=0
Nov 03 09:21:32.452614 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:32.452114    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9" event={"ID":"1224423e-4ddf-4902-80c7-19170efb4b38","Type":"ContainerDied","Data":"bf7662cc3fa8000a9e3178a6454d3618d2612e783aa508c9c80c1dd8b0a01f27"}
Nov 03 09:21:32.452686 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:32.452666    4473 scope.go:122] "RemoveContainer" containerID="bf7662cc3fa8000a9e3178a6454d3618d2612e783aa508c9c80c1dd8b0a01f27"
Nov 03 09:21:32.454444 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:32.454413    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:21:32.454535 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:32.454485    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:21:32.454535 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:32.454520    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:21:33.461566 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:33.461531    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9" event={"ID":"1224423e-4ddf-4902-80c7-19170efb4b38","Type":"ContainerStarted","Data":"94787c360c9ab3106a6463cff2ac3c84e3825c7197e9ce7100d9609777efae6c"}
Nov 03 09:21:33.964101 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:33.964071    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"
Nov 03 09:21:33.964313 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:33.964250    4473 patch_prober.go:29] interesting pod/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 03 09:21:33.964388 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:33.964302    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9" podUID="1224423e-4ddf-4902-80c7-19170efb4b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 03 09:21:34.964205 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:34.964162    4473 patch_prober.go:29] interesting pod/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 03 09:21:34.964702 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:34.964240    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9" podUID="1224423e-4ddf-4902-80c7-19170efb4b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 03 09:21:35.963684 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:35.963649    4473 patch_prober.go:29] interesting pod/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 03 09:21:35.963860 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:35.963718    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9" podUID="1224423e-4ddf-4902-80c7-19170efb4b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 03 09:21:36.963746 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:36.963708    4473 patch_prober.go:29] interesting pod/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 03 09:21:36.964209 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:36.963771    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9" podUID="1224423e-4ddf-4902-80c7-19170efb4b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 03 09:21:37.702921 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:37.702887    4473 scope.go:122] "RemoveContainer" containerID="bb84b2efe161ca4b53b561e03a8d54f19042871f0233bbf499a0a692dc863043"
Nov 03 09:21:37.712421 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:37.712401    4473 scope.go:122] "RemoveContainer" containerID="50bcfb98e558e1ba36121c9b6c4c43bf5018b273f174c63daa1435101433fd8f"
Nov 03 09:21:37.747633 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:37.747613    4473 scope.go:122] "RemoveContainer" containerID="6a24801ebad9e468705c1d09818464d6efed7b4863b37299c524f3ed50b62cbf"
Nov 03 09:21:37.756720 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:37.756688    4473 scope.go:122] "RemoveContainer" containerID="4ac96d55c1d9c2f443010b9736bb9d4835c983ff928fb8c1092a034145cc1929"
Nov 03 09:21:37.765467 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:37.765448    4473 scope.go:122] "RemoveContainer" containerID="715d2bac9d932f0b33bd5f1f6ce80c5e1c2355917337c2126473cce2d4974b37"
Nov 03 09:21:37.963932 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:37.963852    4473 patch_prober.go:29] interesting pod/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 03 09:21:37.963932 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:37.963914    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9" podUID="1224423e-4ddf-4902-80c7-19170efb4b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 03 09:21:38.964023 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:38.963990    4473 patch_prober.go:29] interesting pod/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body=
Nov 03 09:21:38.964435 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:38.964036    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9" podUID="1224423e-4ddf-4902-80c7-19170efb4b38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused"
Nov 03 09:21:39.968238 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:39.968206    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"
Nov 03 09:21:39.968737 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:39.968606    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"
Nov 03 09:21:39.969286 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:39.969267    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"
Nov 03 09:21:46.938138 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:21:46.938104    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:21:47.057533 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:47.057495    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06de7aa4-kh7ng"]
Nov 03 09:21:47.106574 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:47.106547    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06de7aa4-kh7ng"]
Nov 03 09:21:47.870065 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:47.870030    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a250127-84fa-4808-9a98-abd9dd9afe63" path="/var/lib/kubelet/pods/9a250127-84fa-4808-9a98-abd9dd9afe63/volumes"
Nov 03 09:21:49.231328 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:49.231291    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0764e0e9-r7rgm"]
Nov 03 09:21:49.231869 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:49.231854    4473 cpu_manager.go:414] "containerMap: removing container" podUID="94998825-9ad4-4f71-9c54-b4c2a5032c26" containerName="istio-proxy"
Nov 03 09:21:49.231912 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:49.231870    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94998825-9ad4-4f71-9c54-b4c2a5032c26" containerName="istio-proxy"
Nov 03 09:21:49.232006 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:49.231994    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="94998825-9ad4-4f71-9c54-b4c2a5032c26" containerName="istio-proxy"
Nov 03 09:21:49.235367 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:49.235349    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0764e0e9-r7rgm"
Nov 03 09:21:49.296336 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:49.296301    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0764e0e9-r7rgm"]
Nov 03 09:21:49.341675 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:49.341642    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2afd523-012e-4305-9f86-ea7ba8e7d88a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0764e0e9-r7rgm\" (UID: \"d2afd523-012e-4305-9f86-ea7ba8e7d88a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0764e0e9-r7rgm"
Nov 03 09:21:49.341846 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:49.341793    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2afd523-012e-4305-9f86-ea7ba8e7d88a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0764e0e9-r7rgm\" (UID: \"d2afd523-012e-4305-9f86-ea7ba8e7d88a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0764e0e9-r7rgm"
Nov 03 09:21:49.341846 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:49.341809    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:21:49.342000 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:49.341979    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xw9c\" (UniqueName: \"kubernetes.io/projected/d2afd523-012e-4305-9f86-ea7ba8e7d88a-kube-api-access-2xw9c\") pod \"maas-api-revoke-sub-0764e0e9-r7rgm\" (UID: \"d2afd523-012e-4305-9f86-ea7ba8e7d88a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0764e0e9-r7rgm"
Nov 03 09:21:49.342069 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:49.342043    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:21:49.342381 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:49.342365    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:21:49.342653 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:49.342640    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:21:49.443383 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:49.443346    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2afd523-012e-4305-9f86-ea7ba8e7d88a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0764e0e9-r7rgm\" (UID: \"d2afd523-012e-4305-9f86-ea7ba8e7d88a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0764e0e9-r7rgm"
Nov 03 09:21:49.443555 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:49.443469    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xw9c\" (UniqueName: \"kubernetes.io/projected/d2afd523-012e-4305-9f86-ea7ba8e7d88a-kube-api-access-2xw9c\") pod \"maas-api-revoke-sub-0764e0e9-r7rgm\" (UID: \"d2afd523-012e-4305-9f86-ea7ba8e7d88a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0764e0e9-r7rgm"
Nov 03 09:21:49.443555 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:49.443497    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2afd523-012e-4305-9f86-ea7ba8e7d88a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0764e0e9-r7rgm\" (UID: \"d2afd523-012e-4305-9f86-ea7ba8e7d88a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0764e0e9-r7rgm"
Nov 03 09:21:49.444109 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:49.444047    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2afd523-012e-4305-9f86-ea7ba8e7d88a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0764e0e9-r7rgm\" (UID: \"d2afd523-012e-4305-9f86-ea7ba8e7d88a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0764e0e9-r7rgm"
Nov 03 09:21:49.445863 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:49.445840    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2afd523-012e-4305-9f86-ea7ba8e7d88a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0764e0e9-r7rgm\" (UID: \"d2afd523-012e-4305-9f86-ea7ba8e7d88a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0764e0e9-r7rgm"
Nov 03 09:21:49.503001 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:49.502902    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xw9c\" (UniqueName: \"kubernetes.io/projected/d2afd523-012e-4305-9f86-ea7ba8e7d88a-kube-api-access-2xw9c\") pod \"maas-api-revoke-sub-0764e0e9-r7rgm\" (UID: \"d2afd523-012e-4305-9f86-ea7ba8e7d88a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0764e0e9-r7rgm"
Nov 03 09:21:49.547709 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:49.547677    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0764e0e9-r7rgm"
Nov 03 09:21:49.940095 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:21:49.940057    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2afd523_012e_4305_9f86_ea7ba8e7d88a.slice/crio-3be6bf8dc6bace48179af4109c256f49b3697fa6d44bf9b2da7ae34b4ba9ae7a WatchSource:0}: Error finding container 3be6bf8dc6bace48179af4109c256f49b3697fa6d44bf9b2da7ae34b4ba9ae7a: Status 404 returned error can't find the container with id 3be6bf8dc6bace48179af4109c256f49b3697fa6d44bf9b2da7ae34b4ba9ae7a
Nov 03 09:21:49.940241 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:49.940199    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0764e0e9-r7rgm"]
Nov 03 09:21:50.575031 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:50.574303    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0764e0e9-r7rgm" event={"ID":"d2afd523-012e-4305-9f86-ea7ba8e7d88a","Type":"ContainerStarted","Data":"0b85a48546eb6bfb2a93a5a73017556bda2b139936a6ed30a4332b81dba03bd3"}
Nov 03 09:21:50.575031 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:50.574383    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0764e0e9-r7rgm" event={"ID":"d2afd523-012e-4305-9f86-ea7ba8e7d88a","Type":"ContainerStarted","Data":"3be6bf8dc6bace48179af4109c256f49b3697fa6d44bf9b2da7ae34b4ba9ae7a"}
Nov 03 09:21:50.696169 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:50.696111    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0764e0e9-r7rgm" podStartSLOduration=1.6960985910000002 podStartE2EDuration="1.696098591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:21:50.691973298 +0000 UTC m=+2534.961036733" watchObservedRunningTime="2026-11-03 09:21:50.696098591 +0000 UTC m=+2534.965162025"
Nov 03 09:21:51.581774 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:51.581737    4473 generic.go:356] "Generic (PLEG): container finished" podID="d2afd523-012e-4305-9f86-ea7ba8e7d88a" containerID="0b85a48546eb6bfb2a93a5a73017556bda2b139936a6ed30a4332b81dba03bd3" exitCode=0
Nov 03 09:21:51.582219 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:51.581786    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0764e0e9-r7rgm" event={"ID":"d2afd523-012e-4305-9f86-ea7ba8e7d88a","Type":"ContainerDied","Data":"0b85a48546eb6bfb2a93a5a73017556bda2b139936a6ed30a4332b81dba03bd3"}
Nov 03 09:21:52.745087 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:52.745059    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0764e0e9-r7rgm"
Nov 03 09:21:52.891958 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:52.891912    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2afd523-012e-4305-9f86-ea7ba8e7d88a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2afd523-012e-4305-9f86-ea7ba8e7d88a-maas-api-xks-ca\") pod \"d2afd523-012e-4305-9f86-ea7ba8e7d88a\" (UID: \"d2afd523-012e-4305-9f86-ea7ba8e7d88a\") "
Nov 03 09:21:52.892138 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:52.891973    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2afd523-012e-4305-9f86-ea7ba8e7d88a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2afd523-012e-4305-9f86-ea7ba8e7d88a-maas-api-openshift-ca\") pod \"d2afd523-012e-4305-9f86-ea7ba8e7d88a\" (UID: \"d2afd523-012e-4305-9f86-ea7ba8e7d88a\") "
Nov 03 09:21:52.892186 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:52.892139    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2afd523-012e-4305-9f86-ea7ba8e7d88a-kube-api-access-2xw9c\" (UniqueName: \"kubernetes.io/projected/d2afd523-012e-4305-9f86-ea7ba8e7d88a-kube-api-access-2xw9c\") pod \"d2afd523-012e-4305-9f86-ea7ba8e7d88a\" (UID: \"d2afd523-012e-4305-9f86-ea7ba8e7d88a\") "
Nov 03 09:21:52.892437 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:52.892401    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2afd523-012e-4305-9f86-ea7ba8e7d88a-maas-api-openshift-ca" pod "d2afd523-012e-4305-9f86-ea7ba8e7d88a" (UID: "d2afd523-012e-4305-9f86-ea7ba8e7d88a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:21:52.894347 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:52.894314    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2afd523-012e-4305-9f86-ea7ba8e7d88a-maas-api-xks-ca" pod "d2afd523-012e-4305-9f86-ea7ba8e7d88a" (UID: "d2afd523-012e-4305-9f86-ea7ba8e7d88a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:21:52.894462 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:52.894423    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2afd523-012e-4305-9f86-ea7ba8e7d88a-kube-api-access-2xw9c" pod "d2afd523-012e-4305-9f86-ea7ba8e7d88a" (UID: "d2afd523-012e-4305-9f86-ea7ba8e7d88a"). InnerVolumeSpecName "kube-api-access-2xw9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:21:52.993933 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:52.993895    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2afd523-012e-4305-9f86-ea7ba8e7d88a-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:21:52.993933 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:52.993922    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2afd523-012e-4305-9f86-ea7ba8e7d88a-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:21:52.993933 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:52.993933    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xw9c\" (UniqueName: \"kubernetes.io/projected/d2afd523-012e-4305-9f86-ea7ba8e7d88a-kube-api-access-2xw9c\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:21:53.057444 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:53.057410    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c954621c-kxqb2"]
Nov 03 09:21:53.061783 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:53.061758    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c954621c-kxqb2"]
Nov 03 09:21:53.597432 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:53.597401    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0764e0e9-r7rgm" event={"ID":"d2afd523-012e-4305-9f86-ea7ba8e7d88a","Type":"ContainerDied","Data":"3be6bf8dc6bace48179af4109c256f49b3697fa6d44bf9b2da7ae34b4ba9ae7a"}
Nov 03 09:21:53.597615 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:53.597450    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3be6bf8dc6bace48179af4109c256f49b3697fa6d44bf9b2da7ae34b4ba9ae7a"
Nov 03 09:21:53.597615 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:53.597416    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0764e0e9-r7rgm"
Nov 03 09:21:53.870264 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:21:53.870228    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88a1b3ae-3cdd-43e9-b498-34ef27a3ead7" path="/var/lib/kubelet/pods/88a1b3ae-3cdd-43e9-b498-34ef27a3ead7/volumes"
Nov 03 09:22:03.054899 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:03.054843    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fadbc327-trn6s"]
Nov 03 09:22:03.105516 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:03.105480    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fadbc327-trn6s"]
Nov 03 09:22:03.870637 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:03.870596    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24ee886a-7918-4021-a818-0cdea41cf044" path="/var/lib/kubelet/pods/24ee886a-7918-4021-a818-0cdea41cf044/volumes"
Nov 03 09:22:04.963510 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:04.963480    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:22:05.097173 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:05.097141    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-433821/auth-test-model-433821-kserve-b9dc66df-7psmx"]
Nov 03 09:22:05.097747 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:05.097730    4473 cpu_manager.go:414] "containerMap: removing container" podUID="d2afd523-012e-4305-9f86-ea7ba8e7d88a" containerName="revoke-keys"
Nov 03 09:22:05.097827 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:05.097751    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2afd523-012e-4305-9f86-ea7ba8e7d88a" containerName="revoke-keys"
Nov 03 09:22:05.097880 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:05.097862    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2afd523-012e-4305-9f86-ea7ba8e7d88a" containerName="revoke-keys"
Nov 03 09:22:05.100264 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:05.100247    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-433821/auth-test-model-433821-kserve-b9dc66df-7psmx"
Nov 03 09:22:05.159195 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:05.159160    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-433821/auth-test-model-433821-kserve-b9dc66df-7psmx"]
Nov 03 09:22:05.164810 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:05.164781    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e99dc6a6-6192-473b-a7b2-27cc028c773c-dshm\") pod \"auth-test-model-433821-kserve-b9dc66df-7psmx\" (UID: \"e99dc6a6-6192-473b-a7b2-27cc028c773c\") " pod="ai-tenant-e2e-shared-a-w2-433821/auth-test-model-433821-kserve-b9dc66df-7psmx"
Nov 03 09:22:05.164987 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:05.164827    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e99dc6a6-6192-473b-a7b2-27cc028c773c-home\") pod \"auth-test-model-433821-kserve-b9dc66df-7psmx\" (UID: \"e99dc6a6-6192-473b-a7b2-27cc028c773c\") " pod="ai-tenant-e2e-shared-a-w2-433821/auth-test-model-433821-kserve-b9dc66df-7psmx"
Nov 03 09:22:05.164987 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:05.164849    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e99dc6a6-6192-473b-a7b2-27cc028c773c-model-cache\") pod \"auth-test-model-433821-kserve-b9dc66df-7psmx\" (UID: \"e99dc6a6-6192-473b-a7b2-27cc028c773c\") " pod="ai-tenant-e2e-shared-a-w2-433821/auth-test-model-433821-kserve-b9dc66df-7psmx"
Nov 03 09:22:05.165100 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:05.165081    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e99dc6a6-6192-473b-a7b2-27cc028c773c-tls-certs\") pod \"auth-test-model-433821-kserve-b9dc66df-7psmx\" (UID: \"e99dc6a6-6192-473b-a7b2-27cc028c773c\") " pod="ai-tenant-e2e-shared-a-w2-433821/auth-test-model-433821-kserve-b9dc66df-7psmx"
Nov 03 09:22:05.165170 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:05.165135    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e99dc6a6-6192-473b-a7b2-27cc028c773c-tmp-dir\") pod \"auth-test-model-433821-kserve-b9dc66df-7psmx\" (UID: \"e99dc6a6-6192-473b-a7b2-27cc028c773c\") " pod="ai-tenant-e2e-shared-a-w2-433821/auth-test-model-433821-kserve-b9dc66df-7psmx"
Nov 03 09:22:05.165301 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:05.165214    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6xwk\" (UniqueName: \"kubernetes.io/projected/e99dc6a6-6192-473b-a7b2-27cc028c773c-kube-api-access-q6xwk\") pod \"auth-test-model-433821-kserve-b9dc66df-7psmx\" (UID: \"e99dc6a6-6192-473b-a7b2-27cc028c773c\") " pod="ai-tenant-e2e-shared-a-w2-433821/auth-test-model-433821-kserve-b9dc66df-7psmx"
Nov 03 09:22:05.205107 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:05.205062    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-433821\"/\"openshift-service-ca.crt\""
Nov 03 09:22:05.205107 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:05.205096    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-433821\"/\"default-dockercfg-mvdq5\""
Nov 03 09:22:05.205351 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:05.205100    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-433821\"/\"kube-root-ca.crt\""
Nov 03 09:22:05.205351 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:05.205143    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-433821\"/\"auth-test-model-433821-kserve-self-signed-certs\""
Nov 03 09:22:05.266981 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:05.266868    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6xwk\" (UniqueName: \"kubernetes.io/projected/e99dc6a6-6192-473b-a7b2-27cc028c773c-kube-api-access-q6xwk\") pod \"auth-test-model-433821-kserve-b9dc66df-7psmx\" (UID: \"e99dc6a6-6192-473b-a7b2-27cc028c773c\") " pod="ai-tenant-e2e-shared-a-w2-433821/auth-test-model-433821-kserve-b9dc66df-7psmx"
Nov 03 09:22:05.267159 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:05.266978    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e99dc6a6-6192-473b-a7b2-27cc028c773c-dshm\") pod \"auth-test-model-433821-kserve-b9dc66df-7psmx\" (UID: \"e99dc6a6-6192-473b-a7b2-27cc028c773c\") " pod="ai-tenant-e2e-shared-a-w2-433821/auth-test-model-433821-kserve-b9dc66df-7psmx"
Nov 03 09:22:05.267159 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:05.267037    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e99dc6a6-6192-473b-a7b2-27cc028c773c-home\") pod \"auth-test-model-433821-kserve-b9dc66df-7psmx\" (UID: \"e99dc6a6-6192-473b-a7b2-27cc028c773c\") " pod="ai-tenant-e2e-shared-a-w2-433821/auth-test-model-433821-kserve-b9dc66df-7psmx"
Nov 03 09:22:05.267159 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:05.267067    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e99dc6a6-6192-473b-a7b2-27cc028c773c-model-cache\") pod \"auth-test-model-433821-kserve-b9dc66df-7psmx\" (UID: \"e99dc6a6-6192-473b-a7b2-27cc028c773c\") " pod="ai-tenant-e2e-shared-a-w2-433821/auth-test-model-433821-kserve-b9dc66df-7psmx"
Nov 03 09:22:05.267331 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:05.267223    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e99dc6a6-6192-473b-a7b2-27cc028c773c-tls-certs\") pod \"auth-test-model-433821-kserve-b9dc66df-7psmx\" (UID: \"e99dc6a6-6192-473b-a7b2-27cc028c773c\") " pod="ai-tenant-e2e-shared-a-w2-433821/auth-test-model-433821-kserve-b9dc66df-7psmx"
Nov 03 09:22:05.267331 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:05.267291    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e99dc6a6-6192-473b-a7b2-27cc028c773c-tmp-dir\") pod \"auth-test-model-433821-kserve-b9dc66df-7psmx\" (UID: \"e99dc6a6-6192-473b-a7b2-27cc028c773c\") " pod="ai-tenant-e2e-shared-a-w2-433821/auth-test-model-433821-kserve-b9dc66df-7psmx"
Nov 03 09:22:05.267462 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:05.267432    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e99dc6a6-6192-473b-a7b2-27cc028c773c-home\") pod \"auth-test-model-433821-kserve-b9dc66df-7psmx\" (UID: \"e99dc6a6-6192-473b-a7b2-27cc028c773c\") " pod="ai-tenant-e2e-shared-a-w2-433821/auth-test-model-433821-kserve-b9dc66df-7psmx"
Nov 03 09:22:05.267528 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:05.267507    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e99dc6a6-6192-473b-a7b2-27cc028c773c-model-cache\") pod \"auth-test-model-433821-kserve-b9dc66df-7psmx\" (UID: \"e99dc6a6-6192-473b-a7b2-27cc028c773c\") " pod="ai-tenant-e2e-shared-a-w2-433821/auth-test-model-433821-kserve-b9dc66df-7psmx"
Nov 03 09:22:05.267622 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:05.267606    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e99dc6a6-6192-473b-a7b2-27cc028c773c-tmp-dir\") pod \"auth-test-model-433821-kserve-b9dc66df-7psmx\" (UID: \"e99dc6a6-6192-473b-a7b2-27cc028c773c\") " pod="ai-tenant-e2e-shared-a-w2-433821/auth-test-model-433821-kserve-b9dc66df-7psmx"
Nov 03 09:22:05.269423 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:05.269406    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e99dc6a6-6192-473b-a7b2-27cc028c773c-dshm\") pod \"auth-test-model-433821-kserve-b9dc66df-7psmx\" (UID: \"e99dc6a6-6192-473b-a7b2-27cc028c773c\") " pod="ai-tenant-e2e-shared-a-w2-433821/auth-test-model-433821-kserve-b9dc66df-7psmx"
Nov 03 09:22:05.269737 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:05.269721    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e99dc6a6-6192-473b-a7b2-27cc028c773c-tls-certs\") pod \"auth-test-model-433821-kserve-b9dc66df-7psmx\" (UID: \"e99dc6a6-6192-473b-a7b2-27cc028c773c\") " pod="ai-tenant-e2e-shared-a-w2-433821/auth-test-model-433821-kserve-b9dc66df-7psmx"
Nov 03 09:22:05.326362 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:05.326325    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6xwk\" (UniqueName: \"kubernetes.io/projected/e99dc6a6-6192-473b-a7b2-27cc028c773c-kube-api-access-q6xwk\") pod \"auth-test-model-433821-kserve-b9dc66df-7psmx\" (UID: \"e99dc6a6-6192-473b-a7b2-27cc028c773c\") " pod="ai-tenant-e2e-shared-a-w2-433821/auth-test-model-433821-kserve-b9dc66df-7psmx"
Nov 03 09:22:05.412136 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:05.412102    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-433821/auth-test-model-433821-kserve-b9dc66df-7psmx"
Nov 03 09:22:05.805767 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:05.805741    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-433821/auth-test-model-433821-kserve-b9dc66df-7psmx"]
Nov 03 09:22:05.806503 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:22:05.806469    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode99dc6a6_6192_473b_a7b2_27cc028c773c.slice/crio-9f073416ee01496841374079b45e7c75a1a69270c31fdaf47077e11951785f12 WatchSource:0}: Error finding container 9f073416ee01496841374079b45e7c75a1a69270c31fdaf47077e11951785f12: Status 404 returned error can't find the container with id 9f073416ee01496841374079b45e7c75a1a69270c31fdaf47077e11951785f12
Nov 03 09:22:06.702922 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:06.702889    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-433821/auth-test-model-433821-kserve-b9dc66df-7psmx" event={"ID":"e99dc6a6-6192-473b-a7b2-27cc028c773c","Type":"ContainerStarted","Data":"80bec7f1d508d86ac9da68c68e4ef4c8b53f74acc335e590da80ae59ebff4b22"}
Nov 03 09:22:06.702922 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:06.702928    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-433821/auth-test-model-433821-kserve-b9dc66df-7psmx" event={"ID":"e99dc6a6-6192-473b-a7b2-27cc028c773c","Type":"ContainerStarted","Data":"9f073416ee01496841374079b45e7c75a1a69270c31fdaf47077e11951785f12"}
Nov 03 09:22:06.823173 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:06.823118    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-433821/auth-test-model-433821-kserve-b9dc66df-7psmx" podStartSLOduration=1.8231038019999999 podStartE2EDuration="1.823103802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:22:06.819780621 +0000 UTC m=+2551.088844069" watchObservedRunningTime="2026-11-03 09:22:06.823103802 +0000 UTC m=+2551.092167235"
Nov 03 09:22:07.412404 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:07.412372    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-433821/auth-test-model-433821-kserve-b9dc66df-7psmx"
Nov 03 09:22:15.412344 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:15.412310    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-433821/auth-test-model-433821-kserve-b9dc66df-7psmx"
Nov 03 09:22:15.430554 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:15.430524    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-433821/auth-test-model-433821-kserve-b9dc66df-7psmx"
Nov 03 09:22:15.793676 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:15.793600    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-433821/auth-test-model-433821-kserve-b9dc66df-7psmx"
Nov 03 09:22:16.937075 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:22:16.936992    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:22:19.817389 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:19.817362    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-2fc57e-b79446cf5-sfm7v"]
Nov 03 09:22:19.820271 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:19.820251    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2fc57e-b79446cf5-sfm7v"
Nov 03 09:22:19.882144 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:19.882116    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-2fc57e-b79446cf5-sfm7v"]
Nov 03 09:22:19.934978 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:19.924221    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60535aee-30e5-4b8f-a7e3-733e3b933a67-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-2fc57e-b79446cf5-sfm7v\" (UID: \"60535aee-30e5-4b8f-a7e3-733e3b933a67\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-2fc57e-b79446cf5-sfm7v"
Nov 03 09:22:19.936061 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:19.936032    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-896ql\" (UniqueName: \"kubernetes.io/projected/60535aee-30e5-4b8f-a7e3-733e3b933a67-kube-api-access-896ql\") pod \"payload-pre-processing-e2e-explicit-2fc57e-b79446cf5-sfm7v\" (UID: \"60535aee-30e5-4b8f-a7e3-733e3b933a67\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-2fc57e-b79446cf5-sfm7v"
Nov 03 09:22:19.985969 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:19.935678    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-2fc57e-dockercfg-wvg95\""
Nov 03 09:22:20.001806 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:19.935551    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-2fc57e\""
Nov 03 09:22:20.063457 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:20.063413    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60535aee-30e5-4b8f-a7e3-733e3b933a67-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-2fc57e-b79446cf5-sfm7v\" (UID: \"60535aee-30e5-4b8f-a7e3-733e3b933a67\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-2fc57e-b79446cf5-sfm7v"
Nov 03 09:22:20.063797 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:20.063773    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-896ql\" (UniqueName: \"kubernetes.io/projected/60535aee-30e5-4b8f-a7e3-733e3b933a67-kube-api-access-896ql\") pod \"payload-pre-processing-e2e-explicit-2fc57e-b79446cf5-sfm7v\" (UID: \"60535aee-30e5-4b8f-a7e3-733e3b933a67\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-2fc57e-b79446cf5-sfm7v"
Nov 03 09:22:20.065497 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:20.065470    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60535aee-30e5-4b8f-a7e3-733e3b933a67-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-2fc57e-b79446cf5-sfm7v\" (UID: \"60535aee-30e5-4b8f-a7e3-733e3b933a67\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-2fc57e-b79446cf5-sfm7v"
Nov 03 09:22:20.123893 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:20.123863    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-896ql\" (UniqueName: \"kubernetes.io/projected/60535aee-30e5-4b8f-a7e3-733e3b933a67-kube-api-access-896ql\") pod \"payload-pre-processing-e2e-explicit-2fc57e-b79446cf5-sfm7v\" (UID: \"60535aee-30e5-4b8f-a7e3-733e3b933a67\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-2fc57e-b79446cf5-sfm7v"
Nov 03 09:22:20.135725 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:20.135698    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2fc57e-b79446cf5-sfm7v"
Nov 03 09:22:20.516680 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:20.516651    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-2fc57e-b79446cf5-sfm7v"]
Nov 03 09:22:20.517448 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:22:20.517419    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60535aee_30e5_4b8f_a7e3_733e3b933a67.slice/crio-d16ab5554a2d37a40ec9cbd96d57bd8fe73b1ed9c9c206fdac0c0b0c6b4760c1 WatchSource:0}: Error finding container d16ab5554a2d37a40ec9cbd96d57bd8fe73b1ed9c9c206fdac0c0b0c6b4760c1: Status 404 returned error can't find the container with id d16ab5554a2d37a40ec9cbd96d57bd8fe73b1ed9c9c206fdac0c0b0c6b4760c1
Nov 03 09:22:20.821793 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:20.821706    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2fc57e-b79446cf5-sfm7v" event={"ID":"60535aee-30e5-4b8f-a7e3-733e3b933a67","Type":"ContainerStarted","Data":"5c3775e2a7a7f18623a3804833ce3eb7c43c1fa2c3e1ea32cb9598b5e2e365fe"}
Nov 03 09:22:20.821793 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:20.821756    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2fc57e-b79446cf5-sfm7v" event={"ID":"60535aee-30e5-4b8f-a7e3-733e3b933a67","Type":"ContainerStarted","Data":"d16ab5554a2d37a40ec9cbd96d57bd8fe73b1ed9c9c206fdac0c0b0c6b4760c1"}
Nov 03 09:22:20.822391 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:20.821901    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2fc57e-b79446cf5-sfm7v"
Nov 03 09:22:20.940167 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:20.940108    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2fc57e-b79446cf5-sfm7v" podStartSLOduration=1.940095494 podStartE2EDuration="1.940095494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:22:20.937932147 +0000 UTC m=+2565.206995582" watchObservedRunningTime="2026-11-03 09:22:20.940095494 +0000 UTC m=+2565.209158957"
Nov 03 09:22:31.055687 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:31.055653    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-048c3973-7q8d2"]
Nov 03 09:22:31.108103 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:31.108073    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59f25a66-r8zcd"]
Nov 03 09:22:31.112700 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:31.112667    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-048c3973-7q8d2"]
Nov 03 09:22:31.115482 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:31.115457    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59f25a66-r8zcd"]
Nov 03 09:22:31.832540 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:31.832515    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2fc57e-b79446cf5-sfm7v"
Nov 03 09:22:31.873739 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:31.873708    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2206c986-02c0-4e5b-8c37-13ce79cf063a" path="/var/lib/kubelet/pods/2206c986-02c0-4e5b-8c37-13ce79cf063a/volumes"
Nov 03 09:22:31.874179 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:31.874165    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c" path="/var/lib/kubelet/pods/bac2b2a4-3bbc-4729-81c0-1c68fdabcf6c/volumes"
Nov 03 09:22:37.809009 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:37.808973    4473 scope.go:122] "RemoveContainer" containerID="b9e74aa2ab6b0ee6bd5b1e1c63eebdffcec7d526597ac816112f58132d9491f3"
Nov 03 09:22:37.819844 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:37.819823    4473 scope.go:122] "RemoveContainer" containerID="42706b8f40348817bf272da07590a25f11fd73afa8e53ca50560f9cfc0454a99"
Nov 03 09:22:37.829292 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:37.829268    4473 scope.go:122] "RemoveContainer" containerID="80c8612effbeb0fe6f427192f89c3699a110aa349d8fac1ddaf6a66380b7b668"
Nov 03 09:22:37.838790 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:37.838768    4473 scope.go:122] "RemoveContainer" containerID="c98ebdb265367e64dea64a716e1b5814f219a901a93d71e3a05a7e21b4f2df13"
Nov 03 09:22:37.847892 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:37.847868    4473 scope.go:122] "RemoveContainer" containerID="0b5dc3cc87494d123400a7955afb6d36b8a8bb18292c2378f292b536180bab45"
Nov 03 09:22:41.055072 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:41.055038    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e762eaf6-hnk4r"]
Nov 03 09:22:41.062406 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:41.062369    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e762eaf6-hnk4r"]
Nov 03 09:22:41.950590 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:41.950552    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b3ced30-0ba0-45b9-b1fc-49b730f797c1" path="/var/lib/kubelet/pods/1b3ced30-0ba0-45b9-b1fc-49b730f797c1/volumes"
Nov 03 09:22:43.052952 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:43.052909    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7188c7f1-5x2jb"]
Nov 03 09:22:43.103475 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:43.103444    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7188c7f1-5x2jb"]
Nov 03 09:22:43.874937 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:43.873498    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a64fae7c-510c-41a3-8a75-95203daac0bf" path="/var/lib/kubelet/pods/a64fae7c-510c-41a3-8a75-95203daac0bf/volumes"
Nov 03 09:22:45.054765 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:45.054730    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00db84b1-hxwqt"]
Nov 03 09:22:45.060222 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:45.060201    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00db84b1-hxwqt"]
Nov 03 09:22:45.870506 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:45.870462    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfb9ba89-9bc5-4024-b535-deeb4421b4e2" path="/var/lib/kubelet/pods/cfb9ba89-9bc5-4024-b535-deeb4421b4e2/volumes"
Nov 03 09:22:46.963775 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:22:46.963743    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:22:52.146289 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:52.146253    4473 generic.go:356] "Generic (PLEG): container finished" podID="603f6f44-b6ae-4d49-b79d-e73bb6d853a9" containerID="6651d9c20d2ae8db25f249ae0b3c4ed341bc86f976668be7c11ec8b1183a53d1" exitCode=0
Nov 03 09:22:52.146958 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:52.146918    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v" event={"ID":"603f6f44-b6ae-4d49-b79d-e73bb6d853a9","Type":"ContainerDied","Data":"6651d9c20d2ae8db25f249ae0b3c4ed341bc86f976668be7c11ec8b1183a53d1"}
Nov 03 09:22:52.147537 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:52.147516    4473 scope.go:122] "RemoveContainer" containerID="6651d9c20d2ae8db25f249ae0b3c4ed341bc86f976668be7c11ec8b1183a53d1"
Nov 03 09:22:52.150633 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:52.150592    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:22:52.150841 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:52.150819    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:22:52.151008 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:52.150989    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:22:52.462505 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:52.462428    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:22:53.188729 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:53.188674    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v" event={"ID":"603f6f44-b6ae-4d49-b79d-e73bb6d853a9","Type":"ContainerStarted","Data":"c74561f0f1d70f02fae8c7509916bdb5b44ad74f364c8a964f63c3475459b4a1"}
Nov 03 09:22:53.579491 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:53.579405    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"
Nov 03 09:22:53.579853 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:53.579758    4473 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 09:22:53.580087 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:53.580060    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v" podUID="603f6f44-b6ae-4d49-b79d-e73bb6d853a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 09:22:54.571843 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:54.571808    4473 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 09:22:54.572270 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:54.571858    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v" podUID="603f6f44-b6ae-4d49-b79d-e73bb6d853a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 09:22:55.571289 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:55.571253    4473 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 09:22:55.571480 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:55.571321    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v" podUID="603f6f44-b6ae-4d49-b79d-e73bb6d853a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 09:22:56.054994 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:56.054934    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38608a06-qmvgx"]
Nov 03 09:22:56.106578 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:56.106542    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38608a06-qmvgx"]
Nov 03 09:22:56.571417 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:56.571387    4473 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 09:22:56.571598 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:56.571439    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v" podUID="603f6f44-b6ae-4d49-b79d-e73bb6d853a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 09:22:57.571721 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:57.571691    4473 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Nov 03 09:22:57.572147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:57.571742    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v" podUID="603f6f44-b6ae-4d49-b79d-e73bb6d853a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Nov 03 09:22:57.870564 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:57.870535    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b0ba314-f331-4642-8690-5bbcb943f9ed" path="/var/lib/kubelet/pods/5b0ba314-f331-4642-8690-5bbcb943f9ed/volumes"
Nov 03 09:22:58.576475 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:58.576444    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"
Nov 03 09:22:58.576959 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:58.576884    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"
Nov 03 09:22:58.577586 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:22:58.577567    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"
Nov 03 09:23:16.938170 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:23:16.938125    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:23:25.770638 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:25.770608    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:23:37.053688 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:37.053654    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dce7c497-rr48n"]
Nov 03 09:23:37.104403 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:37.104373    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dce7c497-rr48n"]
Nov 03 09:23:37.870424 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:37.870387    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c5d409a-585f-4947-a600-7c504df3321e" path="/var/lib/kubelet/pods/4c5d409a-585f-4947-a600-7c504df3321e/volumes"
Nov 03 09:23:37.893266 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:37.893241    4473 scope.go:122] "RemoveContainer" containerID="6e89a8c8de59433d0d50d85f9ee0acbc0035ff7281de3a8a612b2011ec7b2aef"
Nov 03 09:23:37.904201 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:37.904183    4473 scope.go:122] "RemoveContainer" containerID="32b84428177285af1433941a8200b8e139a3431797c5ce2492d1bea86193e4f9"
Nov 03 09:23:37.913302 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:37.913283    4473 scope.go:122] "RemoveContainer" containerID="68cde13d9a5b2d18d806b006161874cb9fae146633eb0a2b1368a333bba331a4"
Nov 03 09:23:37.922458 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:37.922426    4473 scope.go:122] "RemoveContainer" containerID="107571e02772fc15774f22d4d34b91c90c1e92acb6a957f39057012877932977"
Nov 03 09:23:37.933455 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:37.933433    4473 scope.go:122] "RemoveContainer" containerID="861bf4059b8a3ae883eeed366790a36f43a68d7b302a3edf1600ffdbb935fd77"
Nov 03 09:23:37.942160 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:37.942137    4473 scope.go:122] "RemoveContainer" containerID="a5ee69b7f94061caff00b09744d65fbc3beb8246a7627a72c9490cbce605b709"
Nov 03 09:23:46.756554 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:46.756520    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-465707/auth-test-model-465707-kserve-75c6b79fbf-4v284"]
Nov 03 09:23:46.761926 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:46.761904    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-465707/auth-test-model-465707-kserve-75c6b79fbf-4v284"
Nov 03 09:23:46.819049 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:46.819019    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-465707/auth-test-model-465707-kserve-75c6b79fbf-4v284"]
Nov 03 09:23:46.844727 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:46.844700    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rln2b\" (UniqueName: \"kubernetes.io/projected/9cce93be-a3e6-45db-9a17-3560f3822c2e-kube-api-access-rln2b\") pod \"auth-test-model-465707-kserve-75c6b79fbf-4v284\" (UID: \"9cce93be-a3e6-45db-9a17-3560f3822c2e\") " pod="ai-tenant-e2e-shared-a-w1-465707/auth-test-model-465707-kserve-75c6b79fbf-4v284"
Nov 03 09:23:46.844897 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:46.844859    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cce93be-a3e6-45db-9a17-3560f3822c2e-dshm\") pod \"auth-test-model-465707-kserve-75c6b79fbf-4v284\" (UID: \"9cce93be-a3e6-45db-9a17-3560f3822c2e\") " pod="ai-tenant-e2e-shared-a-w1-465707/auth-test-model-465707-kserve-75c6b79fbf-4v284"
Nov 03 09:23:46.844984 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:46.844930    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cce93be-a3e6-45db-9a17-3560f3822c2e-home\") pod \"auth-test-model-465707-kserve-75c6b79fbf-4v284\" (UID: \"9cce93be-a3e6-45db-9a17-3560f3822c2e\") " pod="ai-tenant-e2e-shared-a-w1-465707/auth-test-model-465707-kserve-75c6b79fbf-4v284"
Nov 03 09:23:46.845087 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:46.845064    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cce93be-a3e6-45db-9a17-3560f3822c2e-model-cache\") pod \"auth-test-model-465707-kserve-75c6b79fbf-4v284\" (UID: \"9cce93be-a3e6-45db-9a17-3560f3822c2e\") " pod="ai-tenant-e2e-shared-a-w1-465707/auth-test-model-465707-kserve-75c6b79fbf-4v284"
Nov 03 09:23:46.845165 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:46.845150    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cce93be-a3e6-45db-9a17-3560f3822c2e-tls-certs\") pod \"auth-test-model-465707-kserve-75c6b79fbf-4v284\" (UID: \"9cce93be-a3e6-45db-9a17-3560f3822c2e\") " pod="ai-tenant-e2e-shared-a-w1-465707/auth-test-model-465707-kserve-75c6b79fbf-4v284"
Nov 03 09:23:46.845312 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:46.845286    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cce93be-a3e6-45db-9a17-3560f3822c2e-tmp-dir\") pod \"auth-test-model-465707-kserve-75c6b79fbf-4v284\" (UID: \"9cce93be-a3e6-45db-9a17-3560f3822c2e\") " pod="ai-tenant-e2e-shared-a-w1-465707/auth-test-model-465707-kserve-75c6b79fbf-4v284"
Nov 03 09:23:46.865493 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:46.865463    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-465707\"/\"kube-root-ca.crt\""
Nov 03 09:23:46.865493 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:46.865469    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-465707\"/\"openshift-service-ca.crt\""
Nov 03 09:23:46.865683 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:46.865529    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-465707\"/\"auth-test-model-465707-kserve-self-signed-certs\""
Nov 03 09:23:46.866125 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:46.866111    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-465707\"/\"default-dockercfg-4gfwp\""
Nov 03 09:23:46.946306 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:46.946225    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cce93be-a3e6-45db-9a17-3560f3822c2e-dshm\") pod \"auth-test-model-465707-kserve-75c6b79fbf-4v284\" (UID: \"9cce93be-a3e6-45db-9a17-3560f3822c2e\") " pod="ai-tenant-e2e-shared-a-w1-465707/auth-test-model-465707-kserve-75c6b79fbf-4v284"
Nov 03 09:23:46.946306 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:46.946299    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cce93be-a3e6-45db-9a17-3560f3822c2e-home\") pod \"auth-test-model-465707-kserve-75c6b79fbf-4v284\" (UID: \"9cce93be-a3e6-45db-9a17-3560f3822c2e\") " pod="ai-tenant-e2e-shared-a-w1-465707/auth-test-model-465707-kserve-75c6b79fbf-4v284"
Nov 03 09:23:46.946569 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:46.946323    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cce93be-a3e6-45db-9a17-3560f3822c2e-model-cache\") pod \"auth-test-model-465707-kserve-75c6b79fbf-4v284\" (UID: \"9cce93be-a3e6-45db-9a17-3560f3822c2e\") " pod="ai-tenant-e2e-shared-a-w1-465707/auth-test-model-465707-kserve-75c6b79fbf-4v284"
Nov 03 09:23:46.946569 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:46.946369    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cce93be-a3e6-45db-9a17-3560f3822c2e-tls-certs\") pod \"auth-test-model-465707-kserve-75c6b79fbf-4v284\" (UID: \"9cce93be-a3e6-45db-9a17-3560f3822c2e\") " pod="ai-tenant-e2e-shared-a-w1-465707/auth-test-model-465707-kserve-75c6b79fbf-4v284"
Nov 03 09:23:46.946787 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:46.946764    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cce93be-a3e6-45db-9a17-3560f3822c2e-home\") pod \"auth-test-model-465707-kserve-75c6b79fbf-4v284\" (UID: \"9cce93be-a3e6-45db-9a17-3560f3822c2e\") " pod="ai-tenant-e2e-shared-a-w1-465707/auth-test-model-465707-kserve-75c6b79fbf-4v284"
Nov 03 09:23:46.946879 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:46.946795    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cce93be-a3e6-45db-9a17-3560f3822c2e-tmp-dir\") pod \"auth-test-model-465707-kserve-75c6b79fbf-4v284\" (UID: \"9cce93be-a3e6-45db-9a17-3560f3822c2e\") " pod="ai-tenant-e2e-shared-a-w1-465707/auth-test-model-465707-kserve-75c6b79fbf-4v284"
Nov 03 09:23:46.946879 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:46.946850    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rln2b\" (UniqueName: \"kubernetes.io/projected/9cce93be-a3e6-45db-9a17-3560f3822c2e-kube-api-access-rln2b\") pod \"auth-test-model-465707-kserve-75c6b79fbf-4v284\" (UID: \"9cce93be-a3e6-45db-9a17-3560f3822c2e\") " pod="ai-tenant-e2e-shared-a-w1-465707/auth-test-model-465707-kserve-75c6b79fbf-4v284"
Nov 03 09:23:46.947252 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:46.947228    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cce93be-a3e6-45db-9a17-3560f3822c2e-tmp-dir\") pod \"auth-test-model-465707-kserve-75c6b79fbf-4v284\" (UID: \"9cce93be-a3e6-45db-9a17-3560f3822c2e\") " pod="ai-tenant-e2e-shared-a-w1-465707/auth-test-model-465707-kserve-75c6b79fbf-4v284"
Nov 03 09:23:46.947765 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:46.947689    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cce93be-a3e6-45db-9a17-3560f3822c2e-model-cache\") pod \"auth-test-model-465707-kserve-75c6b79fbf-4v284\" (UID: \"9cce93be-a3e6-45db-9a17-3560f3822c2e\") " pod="ai-tenant-e2e-shared-a-w1-465707/auth-test-model-465707-kserve-75c6b79fbf-4v284"
Nov 03 09:23:46.949325 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:46.949301    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cce93be-a3e6-45db-9a17-3560f3822c2e-dshm\") pod \"auth-test-model-465707-kserve-75c6b79fbf-4v284\" (UID: \"9cce93be-a3e6-45db-9a17-3560f3822c2e\") " pod="ai-tenant-e2e-shared-a-w1-465707/auth-test-model-465707-kserve-75c6b79fbf-4v284"
Nov 03 09:23:46.949540 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:46.949521    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cce93be-a3e6-45db-9a17-3560f3822c2e-tls-certs\") pod \"auth-test-model-465707-kserve-75c6b79fbf-4v284\" (UID: \"9cce93be-a3e6-45db-9a17-3560f3822c2e\") " pod="ai-tenant-e2e-shared-a-w1-465707/auth-test-model-465707-kserve-75c6b79fbf-4v284"
Nov 03 09:23:46.950206 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:23:46.950182    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:23:47.006796 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:47.006730    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rln2b\" (UniqueName: \"kubernetes.io/projected/9cce93be-a3e6-45db-9a17-3560f3822c2e-kube-api-access-rln2b\") pod \"auth-test-model-465707-kserve-75c6b79fbf-4v284\" (UID: \"9cce93be-a3e6-45db-9a17-3560f3822c2e\") " pod="ai-tenant-e2e-shared-a-w1-465707/auth-test-model-465707-kserve-75c6b79fbf-4v284"
Nov 03 09:23:47.074504 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:47.074473    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-465707/auth-test-model-465707-kserve-75c6b79fbf-4v284"
Nov 03 09:23:47.507400 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:47.507377    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-465707/auth-test-model-465707-kserve-75c6b79fbf-4v284"]
Nov 03 09:23:47.508837 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:23:47.508810    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9cce93be_a3e6_45db_9a17_3560f3822c2e.slice/crio-412eaf800ddcf710df84be3d693842673550dc0db9b4ed272bc35d3cacaaab17 WatchSource:0}: Error finding container 412eaf800ddcf710df84be3d693842673550dc0db9b4ed272bc35d3cacaaab17: Status 404 returned error can't find the container with id 412eaf800ddcf710df84be3d693842673550dc0db9b4ed272bc35d3cacaaab17
Nov 03 09:23:47.584343 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:47.584317    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-465707/auth-test-model-465707-kserve-75c6b79fbf-4v284" event={"ID":"9cce93be-a3e6-45db-9a17-3560f3822c2e","Type":"ContainerStarted","Data":"fc6cacad5619cc79e394867b8fe8a50378b20829082b6b3cbe08882e2c7c4e75"}
Nov 03 09:23:47.584527 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:47.584362    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-465707/auth-test-model-465707-kserve-75c6b79fbf-4v284" event={"ID":"9cce93be-a3e6-45db-9a17-3560f3822c2e","Type":"ContainerStarted","Data":"412eaf800ddcf710df84be3d693842673550dc0db9b4ed272bc35d3cacaaab17"}
Nov 03 09:23:47.703047 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:47.702998    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-465707/auth-test-model-465707-kserve-75c6b79fbf-4v284" podStartSLOduration=1.7029816370000002 podStartE2EDuration="1.702981637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:23:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:23:47.70043333 +0000 UTC m=+2651.969496777" watchObservedRunningTime="2026-11-03 09:23:47.702981637 +0000 UTC m=+2651.972045072"
Nov 03 09:23:48.074766 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:48.074733    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-465707/auth-test-model-465707-kserve-75c6b79fbf-4v284"
Nov 03 09:23:57.075711 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:57.075681    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-465707/auth-test-model-465707-kserve-75c6b79fbf-4v284"
Nov 03 09:23:57.087889 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:57.087862    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-465707/auth-test-model-465707-kserve-75c6b79fbf-4v284"
Nov 03 09:23:57.660357 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:23:57.660327    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-465707/auth-test-model-465707-kserve-75c6b79fbf-4v284"
Nov 03 09:24:02.672078 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:02.672048    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"]
Nov 03 09:24:02.672572 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:02.672273    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q" podUID="9d33af3b-997c-4af3-8e58-fa2b1b78135b" containerName="istio-proxy" containerID="cri-o://d6e116abaf55b444526eb36dbab67f02257d96b726b66ab66132091f61818abb" gracePeriod=30
Nov 03 09:24:02.724554 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:02.724516    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"]
Nov 03 09:24:02.724765 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:02.724744    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr" podUID="3bd912d9-a203-42ab-b4e4-22fd099dd1e5" containerName="istio-proxy" containerID="cri-o://55e1fe8b516abe3ca07578719cf3d75d097688ed575c2bbb0080e308d202a650" gracePeriod=30
Nov 03 09:24:02.727528 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:02.727503    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"]
Nov 03 09:24:02.727742 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:02.727719    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9" podUID="bddb43dc-284b-475c-8f28-d8b320be94b3" containerName="istio-proxy" containerID="cri-o://be36647e9a278d5271fc3c412fd6743fa2b19c1e9979776b925d8596b8514eb4" gracePeriod=30
Nov 03 09:24:02.773255 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:02.773225    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"]
Nov 03 09:24:02.773467 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:02.773445    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb" podUID="cb6f0603-0fa2-4223-b0d6-5411752e25cc" containerName="istio-proxy" containerID="cri-o://fbe4570e2c773b2572cdea384003f88d8058bea1988fcb8201480649614e3bc7" gracePeriod=30
Nov 03 09:24:04.801979 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:04.801910    4473 patch_prober.go:29] interesting pod/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 09:24:04.802367 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:04.801992    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb" podUID="cb6f0603-0fa2-4223-b0d6-5411752e25cc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 09:24:08.059076 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.059003    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"
Nov 03 09:24:08.086429 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.086396    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"
Nov 03 09:24:08.118127 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.118105    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"
Nov 03 09:24:08.120980 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.120961    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"
Nov 03 09:24:08.220350 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.220319    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb6f0603-0fa2-4223-b0d6-5411752e25cc-kube-api-access-zxpd7\" (UniqueName: \"kubernetes.io/projected/cb6f0603-0fa2-4223-b0d6-5411752e25cc-kube-api-access-zxpd7\") pod \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\" (UID: \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\") "
Nov 03 09:24:08.220521 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.220359    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d33af3b-997c-4af3-8e58-fa2b1b78135b-kube-api-access-qm6bs\" (UniqueName: \"kubernetes.io/projected/9d33af3b-997c-4af3-8e58-fa2b1b78135b-kube-api-access-qm6bs\") pod \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\" (UID: \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\") "
Nov 03 09:24:08.220521 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.220397    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d33af3b-997c-4af3-8e58-fa2b1b78135b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d33af3b-997c-4af3-8e58-fa2b1b78135b-istio-podinfo\") pod \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\" (UID: \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\") "
Nov 03 09:24:08.220521 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.220425    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bddb43dc-284b-475c-8f28-d8b320be94b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bddb43dc-284b-475c-8f28-d8b320be94b3-istio-podinfo\") pod \"bddb43dc-284b-475c-8f28-d8b320be94b3\" (UID: \"bddb43dc-284b-475c-8f28-d8b320be94b3\") "
Nov 03 09:24:08.220521 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.220452    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb6f0603-0fa2-4223-b0d6-5411752e25cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb6f0603-0fa2-4223-b0d6-5411752e25cc-workload-socket\") pod \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\" (UID: \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\") "
Nov 03 09:24:08.220521 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.220497    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bddb43dc-284b-475c-8f28-d8b320be94b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bddb43dc-284b-475c-8f28-d8b320be94b3-workload-socket\") pod \"bddb43dc-284b-475c-8f28-d8b320be94b3\" (UID: \"bddb43dc-284b-475c-8f28-d8b320be94b3\") "
Nov 03 09:24:08.221009 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.220583    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d33af3b-997c-4af3-8e58-fa2b1b78135b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d33af3b-997c-4af3-8e58-fa2b1b78135b-istiod-ca-cert\") pod \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\" (UID: \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\") "
Nov 03 09:24:08.221009 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.220629    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-kube-api-access-pnhpz\" (UniqueName: \"kubernetes.io/projected/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-kube-api-access-pnhpz\") pod \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\" (UID: \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\") "
Nov 03 09:24:08.221009 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.220668    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d33af3b-997c-4af3-8e58-fa2b1b78135b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d33af3b-997c-4af3-8e58-fa2b1b78135b-istio-envoy\") pod \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\" (UID: \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\") "
Nov 03 09:24:08.221009 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.220704    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d33af3b-997c-4af3-8e58-fa2b1b78135b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d33af3b-997c-4af3-8e58-fa2b1b78135b-workload-certs\") pod \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\" (UID: \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\") "
Nov 03 09:24:08.221009 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.220744    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bddb43dc-284b-475c-8f28-d8b320be94b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bddb43dc-284b-475c-8f28-d8b320be94b3-istio-envoy\") pod \"bddb43dc-284b-475c-8f28-d8b320be94b3\" (UID: \"bddb43dc-284b-475c-8f28-d8b320be94b3\") "
Nov 03 09:24:08.221009 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.220781    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb6f0603-0fa2-4223-b0d6-5411752e25cc-workload-socket" pod "cb6f0603-0fa2-4223-b0d6-5411752e25cc" (UID: "cb6f0603-0fa2-4223-b0d6-5411752e25cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:24:08.221009 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.220815    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb6f0603-0fa2-4223-b0d6-5411752e25cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb6f0603-0fa2-4223-b0d6-5411752e25cc-workload-certs\") pod \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\" (UID: \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\") "
Nov 03 09:24:08.221009 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.220879    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d33af3b-997c-4af3-8e58-fa2b1b78135b-istio-token\" (UniqueName: \"kubernetes.io/projected/9d33af3b-997c-4af3-8e58-fa2b1b78135b-istio-token\") pod \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\" (UID: \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\") "
Nov 03 09:24:08.221009 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.220936    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb6f0603-0fa2-4223-b0d6-5411752e25cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb6f0603-0fa2-4223-b0d6-5411752e25cc-credential-socket\") pod \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\" (UID: \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\") "
Nov 03 09:24:08.221009 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.221003    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bddb43dc-284b-475c-8f28-d8b320be94b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bddb43dc-284b-475c-8f28-d8b320be94b3-workload-certs\") pod \"bddb43dc-284b-475c-8f28-d8b320be94b3\" (UID: \"bddb43dc-284b-475c-8f28-d8b320be94b3\") "
Nov 03 09:24:08.221676 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.221033    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bddb43dc-284b-475c-8f28-d8b320be94b3-kube-api-access-xkrtg\" (UniqueName: \"kubernetes.io/projected/bddb43dc-284b-475c-8f28-d8b320be94b3-kube-api-access-xkrtg\") pod \"bddb43dc-284b-475c-8f28-d8b320be94b3\" (UID: \"bddb43dc-284b-475c-8f28-d8b320be94b3\") "
Nov 03 09:24:08.221676 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.221059    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bddb43dc-284b-475c-8f28-d8b320be94b3-istio-token\" (UniqueName: \"kubernetes.io/projected/bddb43dc-284b-475c-8f28-d8b320be94b3-istio-token\") pod \"bddb43dc-284b-475c-8f28-d8b320be94b3\" (UID: \"bddb43dc-284b-475c-8f28-d8b320be94b3\") "
Nov 03 09:24:08.221676 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.221065    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb6f0603-0fa2-4223-b0d6-5411752e25cc-workload-certs" pod "cb6f0603-0fa2-4223-b0d6-5411752e25cc" (UID: "cb6f0603-0fa2-4223-b0d6-5411752e25cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:24:08.221676 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.221098    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb6f0603-0fa2-4223-b0d6-5411752e25cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb6f0603-0fa2-4223-b0d6-5411752e25cc-istio-data\") pod \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\" (UID: \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\") "
Nov 03 09:24:08.221676 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.221133    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb6f0603-0fa2-4223-b0d6-5411752e25cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb6f0603-0fa2-4223-b0d6-5411752e25cc-istio-envoy\") pod \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\" (UID: \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\") "
Nov 03 09:24:08.221676 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.221168    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bddb43dc-284b-475c-8f28-d8b320be94b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bddb43dc-284b-475c-8f28-d8b320be94b3-credential-socket\") pod \"bddb43dc-284b-475c-8f28-d8b320be94b3\" (UID: \"bddb43dc-284b-475c-8f28-d8b320be94b3\") "
Nov 03 09:24:08.221676 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.221197    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-istio-token\" (UniqueName: \"kubernetes.io/projected/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-istio-token\") pod \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\" (UID: \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\") "
Nov 03 09:24:08.221676 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.221392    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d33af3b-997c-4af3-8e58-fa2b1b78135b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d33af3b-997c-4af3-8e58-fa2b1b78135b-istio-data\") pod \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\" (UID: \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\") "
Nov 03 09:24:08.221676 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.221421    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d33af3b-997c-4af3-8e58-fa2b1b78135b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d33af3b-997c-4af3-8e58-fa2b1b78135b-credential-socket\") pod \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\" (UID: \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\") "
Nov 03 09:24:08.221676 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.221447    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bddb43dc-284b-475c-8f28-d8b320be94b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bddb43dc-284b-475c-8f28-d8b320be94b3-istiod-ca-cert\") pod \"bddb43dc-284b-475c-8f28-d8b320be94b3\" (UID: \"bddb43dc-284b-475c-8f28-d8b320be94b3\") "
Nov 03 09:24:08.221676 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.221503    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d33af3b-997c-4af3-8e58-fa2b1b78135b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d33af3b-997c-4af3-8e58-fa2b1b78135b-workload-socket\") pod \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\" (UID: \"9d33af3b-997c-4af3-8e58-fa2b1b78135b\") "
Nov 03 09:24:08.221676 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.221552    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-istio-data\") pod \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\" (UID: \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\") "
Nov 03 09:24:08.221676 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.221587    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-workload-certs\") pod \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\" (UID: \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\") "
Nov 03 09:24:08.221676 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.221631    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-istiod-ca-cert\") pod \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\" (UID: \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\") "
Nov 03 09:24:08.222630 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.221692    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb6f0603-0fa2-4223-b0d6-5411752e25cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb6f0603-0fa2-4223-b0d6-5411752e25cc-istiod-ca-cert\") pod \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\" (UID: \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\") "
Nov 03 09:24:08.222630 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.221744    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb6f0603-0fa2-4223-b0d6-5411752e25cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb6f0603-0fa2-4223-b0d6-5411752e25cc-istio-podinfo\") pod \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\" (UID: \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\") "
Nov 03 09:24:08.222630 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.221804    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-credential-socket\") pod \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\" (UID: \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\") "
Nov 03 09:24:08.222630 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.221849    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb6f0603-0fa2-4223-b0d6-5411752e25cc-istio-token\" (UniqueName: \"kubernetes.io/projected/cb6f0603-0fa2-4223-b0d6-5411752e25cc-istio-token\") pod \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\" (UID: \"cb6f0603-0fa2-4223-b0d6-5411752e25cc\") "
Nov 03 09:24:08.222630 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.221878    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-workload-socket\") pod \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\" (UID: \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\") "
Nov 03 09:24:08.222630 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.221923    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bddb43dc-284b-475c-8f28-d8b320be94b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bddb43dc-284b-475c-8f28-d8b320be94b3-istio-data\") pod \"bddb43dc-284b-475c-8f28-d8b320be94b3\" (UID: \"bddb43dc-284b-475c-8f28-d8b320be94b3\") "
Nov 03 09:24:08.222630 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.221999    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-istio-podinfo\") pod \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\" (UID: \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\") "
Nov 03 09:24:08.222630 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.222045    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-istio-envoy\") pod \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\" (UID: \"3bd912d9-a203-42ab-b4e4-22fd099dd1e5\") "
Nov 03 09:24:08.222630 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.221093    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d33af3b-997c-4af3-8e58-fa2b1b78135b-istiod-ca-cert" pod "9d33af3b-997c-4af3-8e58-fa2b1b78135b" (UID: "9d33af3b-997c-4af3-8e58-fa2b1b78135b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:24:08.222630 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.221157    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bddb43dc-284b-475c-8f28-d8b320be94b3-workload-socket" pod "bddb43dc-284b-475c-8f28-d8b320be94b3" (UID: "bddb43dc-284b-475c-8f28-d8b320be94b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:24:08.222630 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.222568    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bddb43dc-284b-475c-8f28-d8b320be94b3-credential-socket" pod "bddb43dc-284b-475c-8f28-d8b320be94b3" (UID: "bddb43dc-284b-475c-8f28-d8b320be94b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:24:08.223404 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.222969    4473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb6f0603-0fa2-4223-b0d6-5411752e25cc-workload-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:08.223404 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.222996    4473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bddb43dc-284b-475c-8f28-d8b320be94b3-credential-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:08.223404 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.223013    4473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb6f0603-0fa2-4223-b0d6-5411752e25cc-workload-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:08.223404 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.223026    4473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bddb43dc-284b-475c-8f28-d8b320be94b3-workload-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:08.223404 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.223041    4473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d33af3b-997c-4af3-8e58-fa2b1b78135b-istiod-ca-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:08.223404 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.222972    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb6f0603-0fa2-4223-b0d6-5411752e25cc-istiod-ca-cert" pod "cb6f0603-0fa2-4223-b0d6-5411752e25cc" (UID: "cb6f0603-0fa2-4223-b0d6-5411752e25cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:24:08.223404 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.223345    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d33af3b-997c-4af3-8e58-fa2b1b78135b-credential-socket" pod "9d33af3b-997c-4af3-8e58-fa2b1b78135b" (UID: "9d33af3b-997c-4af3-8e58-fa2b1b78135b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:24:08.223822 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.223633    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bddb43dc-284b-475c-8f28-d8b320be94b3-istiod-ca-cert" pod "bddb43dc-284b-475c-8f28-d8b320be94b3" (UID: "bddb43dc-284b-475c-8f28-d8b320be94b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:24:08.223822 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.223811    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d33af3b-997c-4af3-8e58-fa2b1b78135b-istio-data" pod "9d33af3b-997c-4af3-8e58-fa2b1b78135b" (UID: "9d33af3b-997c-4af3-8e58-fa2b1b78135b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:24:08.223929 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.223896    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d33af3b-997c-4af3-8e58-fa2b1b78135b-workload-socket" pod "9d33af3b-997c-4af3-8e58-fa2b1b78135b" (UID: "9d33af3b-997c-4af3-8e58-fa2b1b78135b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:24:08.224127 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.224106    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-workload-certs" pod "3bd912d9-a203-42ab-b4e4-22fd099dd1e5" (UID: "3bd912d9-a203-42ab-b4e4-22fd099dd1e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:24:08.224289 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.224265    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-istio-data" pod "3bd912d9-a203-42ab-b4e4-22fd099dd1e5" (UID: "3bd912d9-a203-42ab-b4e4-22fd099dd1e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:24:08.224387 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.224303    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-workload-socket" pod "3bd912d9-a203-42ab-b4e4-22fd099dd1e5" (UID: "3bd912d9-a203-42ab-b4e4-22fd099dd1e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:24:08.224472 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.224450    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-credential-socket" pod "3bd912d9-a203-42ab-b4e4-22fd099dd1e5" (UID: "3bd912d9-a203-42ab-b4e4-22fd099dd1e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:24:08.224588 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.221303    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bddb43dc-284b-475c-8f28-d8b320be94b3-workload-certs" pod "bddb43dc-284b-475c-8f28-d8b320be94b3" (UID: "bddb43dc-284b-475c-8f28-d8b320be94b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:24:08.224588 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.221377    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d33af3b-997c-4af3-8e58-fa2b1b78135b-workload-certs" pod "9d33af3b-997c-4af3-8e58-fa2b1b78135b" (UID: "9d33af3b-997c-4af3-8e58-fa2b1b78135b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:24:08.224588 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.222198    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb6f0603-0fa2-4223-b0d6-5411752e25cc-credential-socket" pod "cb6f0603-0fa2-4223-b0d6-5411752e25cc" (UID: "cb6f0603-0fa2-4223-b0d6-5411752e25cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:24:08.224588 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.224526    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bddb43dc-284b-475c-8f28-d8b320be94b3-istio-data" pod "bddb43dc-284b-475c-8f28-d8b320be94b3" (UID: "bddb43dc-284b-475c-8f28-d8b320be94b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:24:08.226097 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.224680    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb6f0603-0fa2-4223-b0d6-5411752e25cc-istio-data" pod "cb6f0603-0fa2-4223-b0d6-5411752e25cc" (UID: "cb6f0603-0fa2-4223-b0d6-5411752e25cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:24:08.226097 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.224889    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d33af3b-997c-4af3-8e58-fa2b1b78135b-istio-podinfo" pod "9d33af3b-997c-4af3-8e58-fa2b1b78135b" (UID: "9d33af3b-997c-4af3-8e58-fa2b1b78135b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 09:24:08.226097 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.224932    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d33af3b-997c-4af3-8e58-fa2b1b78135b-kube-api-access-qm6bs" pod "9d33af3b-997c-4af3-8e58-fa2b1b78135b" (UID: "9d33af3b-997c-4af3-8e58-fa2b1b78135b"). InnerVolumeSpecName "kube-api-access-qm6bs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:24:08.226097 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.225017    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb6f0603-0fa2-4223-b0d6-5411752e25cc-kube-api-access-zxpd7" pod "cb6f0603-0fa2-4223-b0d6-5411752e25cc" (UID: "cb6f0603-0fa2-4223-b0d6-5411752e25cc"). InnerVolumeSpecName "kube-api-access-zxpd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:24:08.226097 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.225273    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bddb43dc-284b-475c-8f28-d8b320be94b3-istio-podinfo" pod "bddb43dc-284b-475c-8f28-d8b320be94b3" (UID: "bddb43dc-284b-475c-8f28-d8b320be94b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 09:24:08.226097 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.225410    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-istiod-ca-cert" pod "3bd912d9-a203-42ab-b4e4-22fd099dd1e5" (UID: "3bd912d9-a203-42ab-b4e4-22fd099dd1e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:24:08.226097 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.225491    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-kube-api-access-pnhpz" pod "3bd912d9-a203-42ab-b4e4-22fd099dd1e5" (UID: "3bd912d9-a203-42ab-b4e4-22fd099dd1e5"). InnerVolumeSpecName "kube-api-access-pnhpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:24:08.226097 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.225961    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d33af3b-997c-4af3-8e58-fa2b1b78135b-istio-envoy" pod "9d33af3b-997c-4af3-8e58-fa2b1b78135b" (UID: "9d33af3b-997c-4af3-8e58-fa2b1b78135b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:24:08.227313 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.227288    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb6f0603-0fa2-4223-b0d6-5411752e25cc-istio-podinfo" pod "cb6f0603-0fa2-4223-b0d6-5411752e25cc" (UID: "cb6f0603-0fa2-4223-b0d6-5411752e25cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 09:24:08.227647 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.227528    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bddb43dc-284b-475c-8f28-d8b320be94b3-istio-token" pod "bddb43dc-284b-475c-8f28-d8b320be94b3" (UID: "bddb43dc-284b-475c-8f28-d8b320be94b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:24:08.227647 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.227602    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-istio-podinfo" pod "3bd912d9-a203-42ab-b4e4-22fd099dd1e5" (UID: "3bd912d9-a203-42ab-b4e4-22fd099dd1e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 09:24:08.227647 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.227617    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb6f0603-0fa2-4223-b0d6-5411752e25cc-istio-token" pod "cb6f0603-0fa2-4223-b0d6-5411752e25cc" (UID: "cb6f0603-0fa2-4223-b0d6-5411752e25cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:24:08.228084 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.227687    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bddb43dc-284b-475c-8f28-d8b320be94b3-istio-envoy" pod "bddb43dc-284b-475c-8f28-d8b320be94b3" (UID: "bddb43dc-284b-475c-8f28-d8b320be94b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:24:08.228084 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.227746    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d33af3b-997c-4af3-8e58-fa2b1b78135b-istio-token" pod "9d33af3b-997c-4af3-8e58-fa2b1b78135b" (UID: "9d33af3b-997c-4af3-8e58-fa2b1b78135b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:24:08.228084 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.227792    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bddb43dc-284b-475c-8f28-d8b320be94b3-kube-api-access-xkrtg" pod "bddb43dc-284b-475c-8f28-d8b320be94b3" (UID: "bddb43dc-284b-475c-8f28-d8b320be94b3"). InnerVolumeSpecName "kube-api-access-xkrtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:24:08.228334 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.228299    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-istio-token" pod "3bd912d9-a203-42ab-b4e4-22fd099dd1e5" (UID: "3bd912d9-a203-42ab-b4e4-22fd099dd1e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:24:08.229173 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.229116    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb6f0603-0fa2-4223-b0d6-5411752e25cc-istio-envoy" pod "cb6f0603-0fa2-4223-b0d6-5411752e25cc" (UID: "cb6f0603-0fa2-4223-b0d6-5411752e25cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:24:08.229447 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.229428    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-istio-envoy" pod "3bd912d9-a203-42ab-b4e4-22fd099dd1e5" (UID: "3bd912d9-a203-42ab-b4e4-22fd099dd1e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:24:08.324495 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.324461    4473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-istio-podinfo\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:08.324495 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.324496    4473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-istio-envoy\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:08.324729 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.324511    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxpd7\" (UniqueName: \"kubernetes.io/projected/cb6f0603-0fa2-4223-b0d6-5411752e25cc-kube-api-access-zxpd7\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:08.324729 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.324524    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm6bs\" (UniqueName: \"kubernetes.io/projected/9d33af3b-997c-4af3-8e58-fa2b1b78135b-kube-api-access-qm6bs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:08.324729 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.324541    4473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d33af3b-997c-4af3-8e58-fa2b1b78135b-istio-podinfo\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:08.324729 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.324553    4473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bddb43dc-284b-475c-8f28-d8b320be94b3-istio-podinfo\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:08.324729 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.324565    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnhpz\" (UniqueName: \"kubernetes.io/projected/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-kube-api-access-pnhpz\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:08.324729 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.324574    4473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d33af3b-997c-4af3-8e58-fa2b1b78135b-istio-envoy\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:08.324729 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.324581    4473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d33af3b-997c-4af3-8e58-fa2b1b78135b-workload-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:08.324729 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.324592    4473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bddb43dc-284b-475c-8f28-d8b320be94b3-istio-envoy\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:08.324729 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.324600    4473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d33af3b-997c-4af3-8e58-fa2b1b78135b-istio-token\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:08.324729 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.324608    4473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb6f0603-0fa2-4223-b0d6-5411752e25cc-credential-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:08.324729 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.324617    4473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bddb43dc-284b-475c-8f28-d8b320be94b3-workload-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:08.324729 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.324625    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkrtg\" (UniqueName: \"kubernetes.io/projected/bddb43dc-284b-475c-8f28-d8b320be94b3-kube-api-access-xkrtg\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:08.324729 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.324634    4473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bddb43dc-284b-475c-8f28-d8b320be94b3-istio-token\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:08.324729 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.324643    4473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb6f0603-0fa2-4223-b0d6-5411752e25cc-istio-data\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:08.324729 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.324650    4473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb6f0603-0fa2-4223-b0d6-5411752e25cc-istio-envoy\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:08.324729 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.324658    4473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-istio-token\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:08.324729 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.324665    4473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d33af3b-997c-4af3-8e58-fa2b1b78135b-istio-data\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:08.324729 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.324673    4473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d33af3b-997c-4af3-8e58-fa2b1b78135b-credential-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:08.324729 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.324681    4473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bddb43dc-284b-475c-8f28-d8b320be94b3-istiod-ca-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:08.324729 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.324701    4473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d33af3b-997c-4af3-8e58-fa2b1b78135b-workload-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:08.324729 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.324709    4473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-istio-data\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:08.324729 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.324717    4473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-workload-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:08.324729 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.324726    4473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-istiod-ca-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:08.324729 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.324736    4473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb6f0603-0fa2-4223-b0d6-5411752e25cc-istiod-ca-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:08.325524 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.324744    4473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb6f0603-0fa2-4223-b0d6-5411752e25cc-istio-podinfo\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:08.325524 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.324752    4473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-credential-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:08.325524 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.324760    4473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb6f0603-0fa2-4223-b0d6-5411752e25cc-istio-token\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:08.325524 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.324768    4473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bd912d9-a203-42ab-b4e4-22fd099dd1e5-workload-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:08.325524 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.324776    4473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bddb43dc-284b-475c-8f28-d8b320be94b3-istio-data\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:08.720055 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.720024    4473 generic.go:356] "Generic (PLEG): container finished" podID="9d33af3b-997c-4af3-8e58-fa2b1b78135b" containerID="d6e116abaf55b444526eb36dbab67f02257d96b726b66ab66132091f61818abb" exitCode=0
Nov 03 09:24:08.720193 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.720077    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"
Nov 03 09:24:08.720193 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.720122    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q" event={"ID":"9d33af3b-997c-4af3-8e58-fa2b1b78135b","Type":"ContainerDied","Data":"d6e116abaf55b444526eb36dbab67f02257d96b726b66ab66132091f61818abb"}
Nov 03 09:24:08.720193 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.720179    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q" event={"ID":"9d33af3b-997c-4af3-8e58-fa2b1b78135b","Type":"ContainerDied","Data":"91f1bbea4180b348fbe5aeb296eb54b0639e686df86c120c9c139124cba7ea81"}
Nov 03 09:24:08.720359 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.720207    4473 scope.go:122] "RemoveContainer" containerID="d6e116abaf55b444526eb36dbab67f02257d96b726b66ab66132091f61818abb"
Nov 03 09:24:08.722039 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.722014    4473 generic.go:356] "Generic (PLEG): container finished" podID="bddb43dc-284b-475c-8f28-d8b320be94b3" containerID="be36647e9a278d5271fc3c412fd6743fa2b19c1e9979776b925d8596b8514eb4" exitCode=0
Nov 03 09:24:08.722153 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.722122    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9" event={"ID":"bddb43dc-284b-475c-8f28-d8b320be94b3","Type":"ContainerDied","Data":"be36647e9a278d5271fc3c412fd6743fa2b19c1e9979776b925d8596b8514eb4"}
Nov 03 09:24:08.722211 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.722169    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9" event={"ID":"bddb43dc-284b-475c-8f28-d8b320be94b3","Type":"ContainerDied","Data":"12da7615a492c957750bf260a0212855b29d4ca581c141a4ea0c6988b8e30cc6"}
Nov 03 09:24:08.722276 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.722221    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"
Nov 03 09:24:08.724835 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.724806    4473 generic.go:356] "Generic (PLEG): container finished" podID="3bd912d9-a203-42ab-b4e4-22fd099dd1e5" containerID="55e1fe8b516abe3ca07578719cf3d75d097688ed575c2bbb0080e308d202a650" exitCode=0
Nov 03 09:24:08.724983 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.724939    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr" event={"ID":"3bd912d9-a203-42ab-b4e4-22fd099dd1e5","Type":"ContainerDied","Data":"55e1fe8b516abe3ca07578719cf3d75d097688ed575c2bbb0080e308d202a650"}
Nov 03 09:24:08.725048 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.725033    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr" event={"ID":"3bd912d9-a203-42ab-b4e4-22fd099dd1e5","Type":"ContainerDied","Data":"1fd78c7e7879da40a47cb38d0bd5db89a6da76db9b24ad8408fd87bcf1eea595"}
Nov 03 09:24:08.725146 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.725127    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"
Nov 03 09:24:08.728906 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.728880    4473 generic.go:356] "Generic (PLEG): container finished" podID="cb6f0603-0fa2-4223-b0d6-5411752e25cc" containerID="fbe4570e2c773b2572cdea384003f88d8058bea1988fcb8201480649614e3bc7" exitCode=0
Nov 03 09:24:08.729051 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.728923    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb" event={"ID":"cb6f0603-0fa2-4223-b0d6-5411752e25cc","Type":"ContainerDied","Data":"fbe4570e2c773b2572cdea384003f88d8058bea1988fcb8201480649614e3bc7"}
Nov 03 09:24:08.729051 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.729000    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb" event={"ID":"cb6f0603-0fa2-4223-b0d6-5411752e25cc","Type":"ContainerDied","Data":"766048bd23ab0bb113f201b6074a96726160fd8d8f997b96d89c022d1bd34dda"}
Nov 03 09:24:08.729257 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.729241    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"
Nov 03 09:24:08.747093 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.747070    4473 scope.go:122] "RemoveContainer" containerID="d6e116abaf55b444526eb36dbab67f02257d96b726b66ab66132091f61818abb"
Nov 03 09:24:08.747385 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:24:08.747365    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6e116abaf55b444526eb36dbab67f02257d96b726b66ab66132091f61818abb\": container with ID starting with d6e116abaf55b444526eb36dbab67f02257d96b726b66ab66132091f61818abb not found: ID does not exist" containerID="d6e116abaf55b444526eb36dbab67f02257d96b726b66ab66132091f61818abb"
Nov 03 09:24:08.747427 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.747393    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6e116abaf55b444526eb36dbab67f02257d96b726b66ab66132091f61818abb"} err="failed to get container status \"d6e116abaf55b444526eb36dbab67f02257d96b726b66ab66132091f61818abb\": rpc error: code = NotFound desc = could not find container \"d6e116abaf55b444526eb36dbab67f02257d96b726b66ab66132091f61818abb\": container with ID starting with d6e116abaf55b444526eb36dbab67f02257d96b726b66ab66132091f61818abb not found: ID does not exist"
Nov 03 09:24:08.747427 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.747411    4473 scope.go:122] "RemoveContainer" containerID="be36647e9a278d5271fc3c412fd6743fa2b19c1e9979776b925d8596b8514eb4"
Nov 03 09:24:08.756823 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.756804    4473 scope.go:122] "RemoveContainer" containerID="be36647e9a278d5271fc3c412fd6743fa2b19c1e9979776b925d8596b8514eb4"
Nov 03 09:24:08.757082 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:24:08.757064    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be36647e9a278d5271fc3c412fd6743fa2b19c1e9979776b925d8596b8514eb4\": container with ID starting with be36647e9a278d5271fc3c412fd6743fa2b19c1e9979776b925d8596b8514eb4 not found: ID does not exist" containerID="be36647e9a278d5271fc3c412fd6743fa2b19c1e9979776b925d8596b8514eb4"
Nov 03 09:24:08.757154 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.757089    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be36647e9a278d5271fc3c412fd6743fa2b19c1e9979776b925d8596b8514eb4"} err="failed to get container status \"be36647e9a278d5271fc3c412fd6743fa2b19c1e9979776b925d8596b8514eb4\": rpc error: code = NotFound desc = could not find container \"be36647e9a278d5271fc3c412fd6743fa2b19c1e9979776b925d8596b8514eb4\": container with ID starting with be36647e9a278d5271fc3c412fd6743fa2b19c1e9979776b925d8596b8514eb4 not found: ID does not exist"
Nov 03 09:24:08.757154 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.757107    4473 scope.go:122] "RemoveContainer" containerID="55e1fe8b516abe3ca07578719cf3d75d097688ed575c2bbb0080e308d202a650"
Nov 03 09:24:08.766265 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.766240    4473 scope.go:122] "RemoveContainer" containerID="55e1fe8b516abe3ca07578719cf3d75d097688ed575c2bbb0080e308d202a650"
Nov 03 09:24:08.766574 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:24:08.766556    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55e1fe8b516abe3ca07578719cf3d75d097688ed575c2bbb0080e308d202a650\": container with ID starting with 55e1fe8b516abe3ca07578719cf3d75d097688ed575c2bbb0080e308d202a650 not found: ID does not exist" containerID="55e1fe8b516abe3ca07578719cf3d75d097688ed575c2bbb0080e308d202a650"
Nov 03 09:24:08.766658 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.766585    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55e1fe8b516abe3ca07578719cf3d75d097688ed575c2bbb0080e308d202a650"} err="failed to get container status \"55e1fe8b516abe3ca07578719cf3d75d097688ed575c2bbb0080e308d202a650\": rpc error: code = NotFound desc = could not find container \"55e1fe8b516abe3ca07578719cf3d75d097688ed575c2bbb0080e308d202a650\": container with ID starting with 55e1fe8b516abe3ca07578719cf3d75d097688ed575c2bbb0080e308d202a650 not found: ID does not exist"
Nov 03 09:24:08.766658 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.766608    4473 scope.go:122] "RemoveContainer" containerID="fbe4570e2c773b2572cdea384003f88d8058bea1988fcb8201480649614e3bc7"
Nov 03 09:24:08.778854 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.778837    4473 scope.go:122] "RemoveContainer" containerID="fbe4570e2c773b2572cdea384003f88d8058bea1988fcb8201480649614e3bc7"
Nov 03 09:24:08.779157 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:24:08.779138    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbe4570e2c773b2572cdea384003f88d8058bea1988fcb8201480649614e3bc7\": container with ID starting with fbe4570e2c773b2572cdea384003f88d8058bea1988fcb8201480649614e3bc7 not found: ID does not exist" containerID="fbe4570e2c773b2572cdea384003f88d8058bea1988fcb8201480649614e3bc7"
Nov 03 09:24:08.779235 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.779168    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbe4570e2c773b2572cdea384003f88d8058bea1988fcb8201480649614e3bc7"} err="failed to get container status \"fbe4570e2c773b2572cdea384003f88d8058bea1988fcb8201480649614e3bc7\": rpc error: code = NotFound desc = could not find container \"fbe4570e2c773b2572cdea384003f88d8058bea1988fcb8201480649614e3bc7\": container with ID starting with fbe4570e2c773b2572cdea384003f88d8058bea1988fcb8201480649614e3bc7 not found: ID does not exist"
Nov 03 09:24:08.891565 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.891529    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"]
Nov 03 09:24:08.901415 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:08.901392    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-n9p9q"]
Nov 03 09:24:09.014713 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:09.014641    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"]
Nov 03 09:24:09.022184 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:09.022161    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-njtnb"]
Nov 03 09:24:09.153263 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:09.148598    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"]
Nov 03 09:24:09.158540 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:09.158509    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-pw2pr"]
Nov 03 09:24:09.257595 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:09.257566    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"]
Nov 03 09:24:09.260072 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:09.260053    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-26e861-openshift-default-76b64989c5-plgc9"]
Nov 03 09:24:09.870585 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:09.870556    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bd912d9-a203-42ab-b4e4-22fd099dd1e5" path="/var/lib/kubelet/pods/3bd912d9-a203-42ab-b4e4-22fd099dd1e5/volumes"
Nov 03 09:24:09.871090 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:09.871075    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d33af3b-997c-4af3-8e58-fa2b1b78135b" path="/var/lib/kubelet/pods/9d33af3b-997c-4af3-8e58-fa2b1b78135b/volumes"
Nov 03 09:24:09.871492 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:09.871480    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bddb43dc-284b-475c-8f28-d8b320be94b3" path="/var/lib/kubelet/pods/bddb43dc-284b-475c-8f28-d8b320be94b3/volumes"
Nov 03 09:24:09.872222 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:09.872207    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb6f0603-0fa2-4223-b0d6-5411752e25cc" path="/var/lib/kubelet/pods/cb6f0603-0fa2-4223-b0d6-5411752e25cc/volumes"
Nov 03 09:24:12.369274 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:12.369241    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"]
Nov 03 09:24:12.369750 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:12.369463    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk" podUID="03b06f04-3ec5-4d68-847c-1eb86a519e5f" containerName="istio-proxy" containerID="cri-o://be2af7270385a5614cb849717e3f56f2417f2ecea94a105abc5d5a0f704f84b0" gracePeriod=30
Nov 03 09:24:12.419854 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:12.419825    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"]
Nov 03 09:24:12.420085 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:12.420061    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp" podUID="e7892f9a-7f14-4b08-b5a9-933572249e05" containerName="istio-proxy" containerID="cri-o://fec4304477a974d327a33ea2db85a940312118b559b3131a0101301512a2b9c4" gracePeriod=30
Nov 03 09:24:12.422749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:12.422721    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"]
Nov 03 09:24:12.422986 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:12.422935    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq" podUID="63a764c9-6c46-47e6-a12a-190d5d4707a8" containerName="istio-proxy" containerID="cri-o://706e97692add275f449d0f5ece8eefb2e5c3277802c9ac887a63ea442e93e1c7" gracePeriod=30
Nov 03 09:24:16.937404 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:24:16.937368    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:24:17.674968 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.673591    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:24:17.697600 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.697572    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"
Nov 03 09:24:17.746268 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.746243    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03b06f04-3ec5-4d68-847c-1eb86a519e5f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03b06f04-3ec5-4d68-847c-1eb86a519e5f-istio-data\") pod \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\" (UID: \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\") "
Nov 03 09:24:17.746367 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.746287    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03b06f04-3ec5-4d68-847c-1eb86a519e5f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03b06f04-3ec5-4d68-847c-1eb86a519e5f-istiod-ca-cert\") pod \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\" (UID: \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\") "
Nov 03 09:24:17.746439 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.746371    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03b06f04-3ec5-4d68-847c-1eb86a519e5f-kube-api-access-l7nqt\" (UniqueName: \"kubernetes.io/projected/03b06f04-3ec5-4d68-847c-1eb86a519e5f-kube-api-access-l7nqt\") pod \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\" (UID: \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\") "
Nov 03 09:24:17.746439 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.746397    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03b06f04-3ec5-4d68-847c-1eb86a519e5f-istio-token\" (UniqueName: \"kubernetes.io/projected/03b06f04-3ec5-4d68-847c-1eb86a519e5f-istio-token\") pod \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\" (UID: \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\") "
Nov 03 09:24:17.746439 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.746434    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03b06f04-3ec5-4d68-847c-1eb86a519e5f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b06f04-3ec5-4d68-847c-1eb86a519e5f-credential-socket\") pod \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\" (UID: \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\") "
Nov 03 09:24:17.746601 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.746461    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03b06f04-3ec5-4d68-847c-1eb86a519e5f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b06f04-3ec5-4d68-847c-1eb86a519e5f-workload-socket\") pod \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\" (UID: \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\") "
Nov 03 09:24:17.746601 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.746529    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03b06f04-3ec5-4d68-847c-1eb86a519e5f-istio-data" pod "03b06f04-3ec5-4d68-847c-1eb86a519e5f" (UID: "03b06f04-3ec5-4d68-847c-1eb86a519e5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:24:17.746601 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.746585    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03b06f04-3ec5-4d68-847c-1eb86a519e5f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03b06f04-3ec5-4d68-847c-1eb86a519e5f-istio-podinfo\") pod \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\" (UID: \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\") "
Nov 03 09:24:17.746759 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.746609    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03b06f04-3ec5-4d68-847c-1eb86a519e5f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03b06f04-3ec5-4d68-847c-1eb86a519e5f-workload-certs\") pod \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\" (UID: \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\") "
Nov 03 09:24:17.746759 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.746654    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03b06f04-3ec5-4d68-847c-1eb86a519e5f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03b06f04-3ec5-4d68-847c-1eb86a519e5f-istio-envoy\") pod \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\" (UID: \"03b06f04-3ec5-4d68-847c-1eb86a519e5f\") "
Nov 03 09:24:17.746759 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.746690    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03b06f04-3ec5-4d68-847c-1eb86a519e5f-istiod-ca-cert" pod "03b06f04-3ec5-4d68-847c-1eb86a519e5f" (UID: "03b06f04-3ec5-4d68-847c-1eb86a519e5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:24:17.747423 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.746998    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03b06f04-3ec5-4d68-847c-1eb86a519e5f-workload-socket" pod "03b06f04-3ec5-4d68-847c-1eb86a519e5f" (UID: "03b06f04-3ec5-4d68-847c-1eb86a519e5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:24:17.747423 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.747116    4473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b06f04-3ec5-4d68-847c-1eb86a519e5f-workload-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:17.747423 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.747131    4473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03b06f04-3ec5-4d68-847c-1eb86a519e5f-istio-data\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:17.747423 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.747140    4473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03b06f04-3ec5-4d68-847c-1eb86a519e5f-istiod-ca-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:17.747423 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.747315    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03b06f04-3ec5-4d68-847c-1eb86a519e5f-workload-certs" pod "03b06f04-3ec5-4d68-847c-1eb86a519e5f" (UID: "03b06f04-3ec5-4d68-847c-1eb86a519e5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:24:17.747423 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.747420    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03b06f04-3ec5-4d68-847c-1eb86a519e5f-credential-socket" pod "03b06f04-3ec5-4d68-847c-1eb86a519e5f" (UID: "03b06f04-3ec5-4d68-847c-1eb86a519e5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:24:17.749032 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.749013    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03b06f04-3ec5-4d68-847c-1eb86a519e5f-istio-token" pod "03b06f04-3ec5-4d68-847c-1eb86a519e5f" (UID: "03b06f04-3ec5-4d68-847c-1eb86a519e5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:24:17.749441 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.749424    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03b06f04-3ec5-4d68-847c-1eb86a519e5f-istio-podinfo" pod "03b06f04-3ec5-4d68-847c-1eb86a519e5f" (UID: "03b06f04-3ec5-4d68-847c-1eb86a519e5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 09:24:17.749535 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.749513    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03b06f04-3ec5-4d68-847c-1eb86a519e5f-istio-envoy" pod "03b06f04-3ec5-4d68-847c-1eb86a519e5f" (UID: "03b06f04-3ec5-4d68-847c-1eb86a519e5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:24:17.750437 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.750415    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03b06f04-3ec5-4d68-847c-1eb86a519e5f-kube-api-access-l7nqt" pod "03b06f04-3ec5-4d68-847c-1eb86a519e5f" (UID: "03b06f04-3ec5-4d68-847c-1eb86a519e5f"). InnerVolumeSpecName "kube-api-access-l7nqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:24:17.757999 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.757979    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"
Nov 03 09:24:17.759932 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.759915    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"
Nov 03 09:24:17.791202 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.791177    4473 generic.go:356] "Generic (PLEG): container finished" podID="e7892f9a-7f14-4b08-b5a9-933572249e05" containerID="fec4304477a974d327a33ea2db85a940312118b559b3131a0101301512a2b9c4" exitCode=0
Nov 03 09:24:17.791342 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.791217    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"
Nov 03 09:24:17.791342 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.791258    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp" event={"ID":"e7892f9a-7f14-4b08-b5a9-933572249e05","Type":"ContainerDied","Data":"fec4304477a974d327a33ea2db85a940312118b559b3131a0101301512a2b9c4"}
Nov 03 09:24:17.791342 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.791316    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp" event={"ID":"e7892f9a-7f14-4b08-b5a9-933572249e05","Type":"ContainerDied","Data":"a587de4177b3ee6b4f594ca685849f92993407d2d5e83d9570a3e8783e9f5540"}
Nov 03 09:24:17.791499 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.791349    4473 scope.go:122] "RemoveContainer" containerID="fec4304477a974d327a33ea2db85a940312118b559b3131a0101301512a2b9c4"
Nov 03 09:24:17.792553 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.792531    4473 generic.go:356] "Generic (PLEG): container finished" podID="63a764c9-6c46-47e6-a12a-190d5d4707a8" containerID="706e97692add275f449d0f5ece8eefb2e5c3277802c9ac887a63ea442e93e1c7" exitCode=0
Nov 03 09:24:17.792636 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.792585    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"
Nov 03 09:24:17.792636 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.792616    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq" event={"ID":"63a764c9-6c46-47e6-a12a-190d5d4707a8","Type":"ContainerDied","Data":"706e97692add275f449d0f5ece8eefb2e5c3277802c9ac887a63ea442e93e1c7"}
Nov 03 09:24:17.792762 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.792661    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq" event={"ID":"63a764c9-6c46-47e6-a12a-190d5d4707a8","Type":"ContainerDied","Data":"e4f320287512ed2cdbf0059f653d903545f9f8f8717424d1cc32eba2040a8be1"}
Nov 03 09:24:17.795129 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.795109    4473 generic.go:356] "Generic (PLEG): container finished" podID="03b06f04-3ec5-4d68-847c-1eb86a519e5f" containerID="be2af7270385a5614cb849717e3f56f2417f2ecea94a105abc5d5a0f704f84b0" exitCode=0
Nov 03 09:24:17.795223 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.795136    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk" event={"ID":"03b06f04-3ec5-4d68-847c-1eb86a519e5f","Type":"ContainerDied","Data":"be2af7270385a5614cb849717e3f56f2417f2ecea94a105abc5d5a0f704f84b0"}
Nov 03 09:24:17.795223 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.795161    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"
Nov 03 09:24:17.795223 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.795180    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk" event={"ID":"03b06f04-3ec5-4d68-847c-1eb86a519e5f","Type":"ContainerDied","Data":"5de532b1b5466065f5dc482c4bdc6e462a77ba557cf2f8a2e1c9ef1ba1b4295b"}
Nov 03 09:24:17.801675 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.801660    4473 scope.go:122] "RemoveContainer" containerID="fec4304477a974d327a33ea2db85a940312118b559b3131a0101301512a2b9c4"
Nov 03 09:24:17.801974 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:24:17.801934    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fec4304477a974d327a33ea2db85a940312118b559b3131a0101301512a2b9c4\": container with ID starting with fec4304477a974d327a33ea2db85a940312118b559b3131a0101301512a2b9c4 not found: ID does not exist" containerID="fec4304477a974d327a33ea2db85a940312118b559b3131a0101301512a2b9c4"
Nov 03 09:24:17.802070 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.801983    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fec4304477a974d327a33ea2db85a940312118b559b3131a0101301512a2b9c4"} err="failed to get container status \"fec4304477a974d327a33ea2db85a940312118b559b3131a0101301512a2b9c4\": rpc error: code = NotFound desc = could not find container \"fec4304477a974d327a33ea2db85a940312118b559b3131a0101301512a2b9c4\": container with ID starting with fec4304477a974d327a33ea2db85a940312118b559b3131a0101301512a2b9c4 not found: ID does not exist"
Nov 03 09:24:17.802070 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.802006    4473 scope.go:122] "RemoveContainer" containerID="706e97692add275f449d0f5ece8eefb2e5c3277802c9ac887a63ea442e93e1c7"
Nov 03 09:24:17.811176 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.811159    4473 scope.go:122] "RemoveContainer" containerID="706e97692add275f449d0f5ece8eefb2e5c3277802c9ac887a63ea442e93e1c7"
Nov 03 09:24:17.811447 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:24:17.811429    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"706e97692add275f449d0f5ece8eefb2e5c3277802c9ac887a63ea442e93e1c7\": container with ID starting with 706e97692add275f449d0f5ece8eefb2e5c3277802c9ac887a63ea442e93e1c7 not found: ID does not exist" containerID="706e97692add275f449d0f5ece8eefb2e5c3277802c9ac887a63ea442e93e1c7"
Nov 03 09:24:17.811532 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.811458    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"706e97692add275f449d0f5ece8eefb2e5c3277802c9ac887a63ea442e93e1c7"} err="failed to get container status \"706e97692add275f449d0f5ece8eefb2e5c3277802c9ac887a63ea442e93e1c7\": rpc error: code = NotFound desc = could not find container \"706e97692add275f449d0f5ece8eefb2e5c3277802c9ac887a63ea442e93e1c7\": container with ID starting with 706e97692add275f449d0f5ece8eefb2e5c3277802c9ac887a63ea442e93e1c7 not found: ID does not exist"
Nov 03 09:24:17.811532 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.811486    4473 scope.go:122] "RemoveContainer" containerID="be2af7270385a5614cb849717e3f56f2417f2ecea94a105abc5d5a0f704f84b0"
Nov 03 09:24:17.820672 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.820654    4473 scope.go:122] "RemoveContainer" containerID="be2af7270385a5614cb849717e3f56f2417f2ecea94a105abc5d5a0f704f84b0"
Nov 03 09:24:17.820957 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:24:17.820927    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be2af7270385a5614cb849717e3f56f2417f2ecea94a105abc5d5a0f704f84b0\": container with ID starting with be2af7270385a5614cb849717e3f56f2417f2ecea94a105abc5d5a0f704f84b0 not found: ID does not exist" containerID="be2af7270385a5614cb849717e3f56f2417f2ecea94a105abc5d5a0f704f84b0"
Nov 03 09:24:17.821029 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.820964    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be2af7270385a5614cb849717e3f56f2417f2ecea94a105abc5d5a0f704f84b0"} err="failed to get container status \"be2af7270385a5614cb849717e3f56f2417f2ecea94a105abc5d5a0f704f84b0\": rpc error: code = NotFound desc = could not find container \"be2af7270385a5614cb849717e3f56f2417f2ecea94a105abc5d5a0f704f84b0\": container with ID starting with be2af7270385a5614cb849717e3f56f2417f2ecea94a105abc5d5a0f704f84b0 not found: ID does not exist"
Nov 03 09:24:17.848160 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.848133    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7892f9a-7f14-4b08-b5a9-933572249e05-kube-api-access-p6sdz\" (UniqueName: \"kubernetes.io/projected/e7892f9a-7f14-4b08-b5a9-933572249e05-kube-api-access-p6sdz\") pod \"e7892f9a-7f14-4b08-b5a9-933572249e05\" (UID: \"e7892f9a-7f14-4b08-b5a9-933572249e05\") "
Nov 03 09:24:17.848299 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.848182    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7892f9a-7f14-4b08-b5a9-933572249e05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7892f9a-7f14-4b08-b5a9-933572249e05-workload-certs\") pod \"e7892f9a-7f14-4b08-b5a9-933572249e05\" (UID: \"e7892f9a-7f14-4b08-b5a9-933572249e05\") "
Nov 03 09:24:17.848299 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.848214    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63a764c9-6c46-47e6-a12a-190d5d4707a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63a764c9-6c46-47e6-a12a-190d5d4707a8-workload-certs\") pod \"63a764c9-6c46-47e6-a12a-190d5d4707a8\" (UID: \"63a764c9-6c46-47e6-a12a-190d5d4707a8\") "
Nov 03 09:24:17.848299 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.848245    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63a764c9-6c46-47e6-a12a-190d5d4707a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63a764c9-6c46-47e6-a12a-190d5d4707a8-istio-data\") pod \"63a764c9-6c46-47e6-a12a-190d5d4707a8\" (UID: \"63a764c9-6c46-47e6-a12a-190d5d4707a8\") "
Nov 03 09:24:17.848299 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.848287    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63a764c9-6c46-47e6-a12a-190d5d4707a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63a764c9-6c46-47e6-a12a-190d5d4707a8-workload-socket\") pod \"63a764c9-6c46-47e6-a12a-190d5d4707a8\" (UID: \"63a764c9-6c46-47e6-a12a-190d5d4707a8\") "
Nov 03 09:24:17.848510 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.848317    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63a764c9-6c46-47e6-a12a-190d5d4707a8-kube-api-access-x9st4\" (UniqueName: \"kubernetes.io/projected/63a764c9-6c46-47e6-a12a-190d5d4707a8-kube-api-access-x9st4\") pod \"63a764c9-6c46-47e6-a12a-190d5d4707a8\" (UID: \"63a764c9-6c46-47e6-a12a-190d5d4707a8\") "
Nov 03 09:24:17.848510 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.848387    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63a764c9-6c46-47e6-a12a-190d5d4707a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63a764c9-6c46-47e6-a12a-190d5d4707a8-istiod-ca-cert\") pod \"63a764c9-6c46-47e6-a12a-190d5d4707a8\" (UID: \"63a764c9-6c46-47e6-a12a-190d5d4707a8\") "
Nov 03 09:24:17.848510 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.848419    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7892f9a-7f14-4b08-b5a9-933572249e05-istio-token\" (UniqueName: \"kubernetes.io/projected/e7892f9a-7f14-4b08-b5a9-933572249e05-istio-token\") pod \"e7892f9a-7f14-4b08-b5a9-933572249e05\" (UID: \"e7892f9a-7f14-4b08-b5a9-933572249e05\") "
Nov 03 09:24:17.848510 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.848455    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63a764c9-6c46-47e6-a12a-190d5d4707a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63a764c9-6c46-47e6-a12a-190d5d4707a8-istio-envoy\") pod \"63a764c9-6c46-47e6-a12a-190d5d4707a8\" (UID: \"63a764c9-6c46-47e6-a12a-190d5d4707a8\") "
Nov 03 09:24:17.848510 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.848492    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7892f9a-7f14-4b08-b5a9-933572249e05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7892f9a-7f14-4b08-b5a9-933572249e05-istio-envoy\") pod \"e7892f9a-7f14-4b08-b5a9-933572249e05\" (UID: \"e7892f9a-7f14-4b08-b5a9-933572249e05\") "
Nov 03 09:24:17.848759 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.848515    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7892f9a-7f14-4b08-b5a9-933572249e05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7892f9a-7f14-4b08-b5a9-933572249e05-istio-data\") pod \"e7892f9a-7f14-4b08-b5a9-933572249e05\" (UID: \"e7892f9a-7f14-4b08-b5a9-933572249e05\") "
Nov 03 09:24:17.848759 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.848564    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7892f9a-7f14-4b08-b5a9-933572249e05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7892f9a-7f14-4b08-b5a9-933572249e05-credential-socket\") pod \"e7892f9a-7f14-4b08-b5a9-933572249e05\" (UID: \"e7892f9a-7f14-4b08-b5a9-933572249e05\") "
Nov 03 09:24:17.848759 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.848641    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63a764c9-6c46-47e6-a12a-190d5d4707a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63a764c9-6c46-47e6-a12a-190d5d4707a8-istio-podinfo\") pod \"63a764c9-6c46-47e6-a12a-190d5d4707a8\" (UID: \"63a764c9-6c46-47e6-a12a-190d5d4707a8\") "
Nov 03 09:24:17.848759 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.848707    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63a764c9-6c46-47e6-a12a-190d5d4707a8-istio-token\" (UniqueName: \"kubernetes.io/projected/63a764c9-6c46-47e6-a12a-190d5d4707a8-istio-token\") pod \"63a764c9-6c46-47e6-a12a-190d5d4707a8\" (UID: \"63a764c9-6c46-47e6-a12a-190d5d4707a8\") "
Nov 03 09:24:17.848759 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.848744    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7892f9a-7f14-4b08-b5a9-933572249e05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7892f9a-7f14-4b08-b5a9-933572249e05-istiod-ca-cert\") pod \"e7892f9a-7f14-4b08-b5a9-933572249e05\" (UID: \"e7892f9a-7f14-4b08-b5a9-933572249e05\") "
Nov 03 09:24:17.849026 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.848797    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63a764c9-6c46-47e6-a12a-190d5d4707a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63a764c9-6c46-47e6-a12a-190d5d4707a8-credential-socket\") pod \"63a764c9-6c46-47e6-a12a-190d5d4707a8\" (UID: \"63a764c9-6c46-47e6-a12a-190d5d4707a8\") "
Nov 03 09:24:17.849026 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.848815    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7892f9a-7f14-4b08-b5a9-933572249e05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7892f9a-7f14-4b08-b5a9-933572249e05-workload-socket\") pod \"e7892f9a-7f14-4b08-b5a9-933572249e05\" (UID: \"e7892f9a-7f14-4b08-b5a9-933572249e05\") "
Nov 03 09:24:17.849026 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.848839    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7892f9a-7f14-4b08-b5a9-933572249e05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7892f9a-7f14-4b08-b5a9-933572249e05-istio-podinfo\") pod \"e7892f9a-7f14-4b08-b5a9-933572249e05\" (UID: \"e7892f9a-7f14-4b08-b5a9-933572249e05\") "
Nov 03 09:24:17.849384 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.849365    4473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03b06f04-3ec5-4d68-847c-1eb86a519e5f-istio-podinfo\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:17.849462 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.849392    4473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03b06f04-3ec5-4d68-847c-1eb86a519e5f-workload-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:17.849462 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.849406    4473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03b06f04-3ec5-4d68-847c-1eb86a519e5f-istio-envoy\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:17.849462 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.849423    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7nqt\" (UniqueName: \"kubernetes.io/projected/03b06f04-3ec5-4d68-847c-1eb86a519e5f-kube-api-access-l7nqt\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:17.849462 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.849440    4473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03b06f04-3ec5-4d68-847c-1eb86a519e5f-istio-token\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:17.849462 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.849452    4473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b06f04-3ec5-4d68-847c-1eb86a519e5f-credential-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:17.852252 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.850122    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7892f9a-7f14-4b08-b5a9-933572249e05-istiod-ca-cert" pod "e7892f9a-7f14-4b08-b5a9-933572249e05" (UID: "e7892f9a-7f14-4b08-b5a9-933572249e05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:24:17.852252 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.850139    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7892f9a-7f14-4b08-b5a9-933572249e05-workload-certs" pod "e7892f9a-7f14-4b08-b5a9-933572249e05" (UID: "e7892f9a-7f14-4b08-b5a9-933572249e05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:24:17.852252 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.850528    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7892f9a-7f14-4b08-b5a9-933572249e05-istio-data" pod "e7892f9a-7f14-4b08-b5a9-933572249e05" (UID: "e7892f9a-7f14-4b08-b5a9-933572249e05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:24:17.852252 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.850679    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63a764c9-6c46-47e6-a12a-190d5d4707a8-istio-data" pod "63a764c9-6c46-47e6-a12a-190d5d4707a8" (UID: "63a764c9-6c46-47e6-a12a-190d5d4707a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:24:17.852252 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.850851    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63a764c9-6c46-47e6-a12a-190d5d4707a8-workload-certs" pod "63a764c9-6c46-47e6-a12a-190d5d4707a8" (UID: "63a764c9-6c46-47e6-a12a-190d5d4707a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:24:17.852252 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.850959    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7892f9a-7f14-4b08-b5a9-933572249e05-workload-socket" pod "e7892f9a-7f14-4b08-b5a9-933572249e05" (UID: "e7892f9a-7f14-4b08-b5a9-933572249e05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:24:17.852252 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.851046    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7892f9a-7f14-4b08-b5a9-933572249e05-credential-socket" pod "e7892f9a-7f14-4b08-b5a9-933572249e05" (UID: "e7892f9a-7f14-4b08-b5a9-933572249e05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:24:17.852252 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.851308    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63a764c9-6c46-47e6-a12a-190d5d4707a8-istiod-ca-cert" pod "63a764c9-6c46-47e6-a12a-190d5d4707a8" (UID: "63a764c9-6c46-47e6-a12a-190d5d4707a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:24:17.852252 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.851511    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63a764c9-6c46-47e6-a12a-190d5d4707a8-workload-socket" pod "63a764c9-6c46-47e6-a12a-190d5d4707a8" (UID: "63a764c9-6c46-47e6-a12a-190d5d4707a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:24:17.852252 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.852227    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63a764c9-6c46-47e6-a12a-190d5d4707a8-credential-socket" pod "63a764c9-6c46-47e6-a12a-190d5d4707a8" (UID: "63a764c9-6c46-47e6-a12a-190d5d4707a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:24:17.853557 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.853531    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7892f9a-7f14-4b08-b5a9-933572249e05-istio-podinfo" pod "e7892f9a-7f14-4b08-b5a9-933572249e05" (UID: "e7892f9a-7f14-4b08-b5a9-933572249e05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 09:24:17.853800 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.853775    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63a764c9-6c46-47e6-a12a-190d5d4707a8-istio-podinfo" pod "63a764c9-6c46-47e6-a12a-190d5d4707a8" (UID: "63a764c9-6c46-47e6-a12a-190d5d4707a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 09:24:17.854050 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.854020    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63a764c9-6c46-47e6-a12a-190d5d4707a8-istio-envoy" pod "63a764c9-6c46-47e6-a12a-190d5d4707a8" (UID: "63a764c9-6c46-47e6-a12a-190d5d4707a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:24:17.858215 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.858116    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63a764c9-6c46-47e6-a12a-190d5d4707a8-istio-token" pod "63a764c9-6c46-47e6-a12a-190d5d4707a8" (UID: "63a764c9-6c46-47e6-a12a-190d5d4707a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:24:17.858215 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.858136    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7892f9a-7f14-4b08-b5a9-933572249e05-istio-envoy" pod "e7892f9a-7f14-4b08-b5a9-933572249e05" (UID: "e7892f9a-7f14-4b08-b5a9-933572249e05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:24:17.858215 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.858143    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7892f9a-7f14-4b08-b5a9-933572249e05-kube-api-access-p6sdz" pod "e7892f9a-7f14-4b08-b5a9-933572249e05" (UID: "e7892f9a-7f14-4b08-b5a9-933572249e05"). InnerVolumeSpecName "kube-api-access-p6sdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:24:17.858215 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.858153    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7892f9a-7f14-4b08-b5a9-933572249e05-istio-token" pod "e7892f9a-7f14-4b08-b5a9-933572249e05" (UID: "e7892f9a-7f14-4b08-b5a9-933572249e05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:24:17.858215 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.858183    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63a764c9-6c46-47e6-a12a-190d5d4707a8-kube-api-access-x9st4" pod "63a764c9-6c46-47e6-a12a-190d5d4707a8" (UID: "63a764c9-6c46-47e6-a12a-190d5d4707a8"). InnerVolumeSpecName "kube-api-access-x9st4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:24:17.950651 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.950578    4473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7892f9a-7f14-4b08-b5a9-933572249e05-credential-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:17.950651 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.950604    4473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63a764c9-6c46-47e6-a12a-190d5d4707a8-istio-podinfo\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:17.950651 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.950615    4473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63a764c9-6c46-47e6-a12a-190d5d4707a8-istio-token\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:17.950651 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.950624    4473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7892f9a-7f14-4b08-b5a9-933572249e05-istiod-ca-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:17.950651 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.950632    4473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63a764c9-6c46-47e6-a12a-190d5d4707a8-credential-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:17.950651 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.950640    4473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7892f9a-7f14-4b08-b5a9-933572249e05-workload-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:17.950651 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.950648    4473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7892f9a-7f14-4b08-b5a9-933572249e05-istio-podinfo\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:17.950651 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.950657    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6sdz\" (UniqueName: \"kubernetes.io/projected/e7892f9a-7f14-4b08-b5a9-933572249e05-kube-api-access-p6sdz\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:17.951302 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.950667    4473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7892f9a-7f14-4b08-b5a9-933572249e05-workload-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:17.951302 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.950675    4473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63a764c9-6c46-47e6-a12a-190d5d4707a8-workload-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:17.951302 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.950682    4473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63a764c9-6c46-47e6-a12a-190d5d4707a8-istio-data\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:17.951302 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.950690    4473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63a764c9-6c46-47e6-a12a-190d5d4707a8-workload-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:17.951302 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.950699    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9st4\" (UniqueName: \"kubernetes.io/projected/63a764c9-6c46-47e6-a12a-190d5d4707a8-kube-api-access-x9st4\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:17.951302 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.950707    4473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63a764c9-6c46-47e6-a12a-190d5d4707a8-istiod-ca-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:17.951302 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.950715    4473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7892f9a-7f14-4b08-b5a9-933572249e05-istio-token\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:17.951302 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.950724    4473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63a764c9-6c46-47e6-a12a-190d5d4707a8-istio-envoy\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:17.951302 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.950732    4473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7892f9a-7f14-4b08-b5a9-933572249e05-istio-envoy\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:17.951302 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:17.950739    4473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7892f9a-7f14-4b08-b5a9-933572249e05-istio-data\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:24:18.057910 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:18.057841    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"]
Nov 03 09:24:18.060953 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:18.060916    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-mzrrk"]
Nov 03 09:24:18.292105 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:18.292019    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"]
Nov 03 09:24:18.298212 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:18.298182    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-tz9kp"]
Nov 03 09:24:18.411561 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:18.411527    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"]
Nov 03 09:24:18.417825 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:18.417798    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-910d50-openshift-default-7cf965dcb7-d99bq"]
Nov 03 09:24:19.870396 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:19.870359    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03b06f04-3ec5-4d68-847c-1eb86a519e5f" path="/var/lib/kubelet/pods/03b06f04-3ec5-4d68-847c-1eb86a519e5f/volumes"
Nov 03 09:24:19.871003 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:19.870984    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63a764c9-6c46-47e6-a12a-190d5d4707a8" path="/var/lib/kubelet/pods/63a764c9-6c46-47e6-a12a-190d5d4707a8/volumes"
Nov 03 09:24:19.871457 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:19.871439    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7892f9a-7f14-4b08-b5a9-933572249e05" path="/var/lib/kubelet/pods/e7892f9a-7f14-4b08-b5a9-933572249e05/volumes"
Nov 03 09:24:33.188205 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.188166    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-f9b517/auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w"]
Nov 03 09:24:33.189308 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.189281    4473 cpu_manager.go:414] "containerMap: removing container" podUID="3bd912d9-a203-42ab-b4e4-22fd099dd1e5" containerName="istio-proxy"
Nov 03 09:24:33.189308 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.189308    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bd912d9-a203-42ab-b4e4-22fd099dd1e5" containerName="istio-proxy"
Nov 03 09:24:33.189490 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.189339    4473 cpu_manager.go:414] "containerMap: removing container" podUID="63a764c9-6c46-47e6-a12a-190d5d4707a8" containerName="istio-proxy"
Nov 03 09:24:33.189490 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.189347    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63a764c9-6c46-47e6-a12a-190d5d4707a8" containerName="istio-proxy"
Nov 03 09:24:33.189490 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.189372    4473 cpu_manager.go:414] "containerMap: removing container" podUID="bddb43dc-284b-475c-8f28-d8b320be94b3" containerName="istio-proxy"
Nov 03 09:24:33.189490 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.189379    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bddb43dc-284b-475c-8f28-d8b320be94b3" containerName="istio-proxy"
Nov 03 09:24:33.189490 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.189413    4473 cpu_manager.go:414] "containerMap: removing container" podUID="cb6f0603-0fa2-4223-b0d6-5411752e25cc" containerName="istio-proxy"
Nov 03 09:24:33.189490 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.189420    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb6f0603-0fa2-4223-b0d6-5411752e25cc" containerName="istio-proxy"
Nov 03 09:24:33.189490 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.189450    4473 cpu_manager.go:414] "containerMap: removing container" podUID="9d33af3b-997c-4af3-8e58-fa2b1b78135b" containerName="istio-proxy"
Nov 03 09:24:33.189490 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.189457    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d33af3b-997c-4af3-8e58-fa2b1b78135b" containerName="istio-proxy"
Nov 03 09:24:33.189490 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.189478    4473 cpu_manager.go:414] "containerMap: removing container" podUID="e7892f9a-7f14-4b08-b5a9-933572249e05" containerName="istio-proxy"
Nov 03 09:24:33.189490 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.189485    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7892f9a-7f14-4b08-b5a9-933572249e05" containerName="istio-proxy"
Nov 03 09:24:33.189969 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.189508    4473 cpu_manager.go:414] "containerMap: removing container" podUID="03b06f04-3ec5-4d68-847c-1eb86a519e5f" containerName="istio-proxy"
Nov 03 09:24:33.189969 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.189515    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03b06f04-3ec5-4d68-847c-1eb86a519e5f" containerName="istio-proxy"
Nov 03 09:24:33.189969 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.189775    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bd912d9-a203-42ab-b4e4-22fd099dd1e5" containerName="istio-proxy"
Nov 03 09:24:33.189969 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.189804    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="63a764c9-6c46-47e6-a12a-190d5d4707a8" containerName="istio-proxy"
Nov 03 09:24:33.189969 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.189814    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="bddb43dc-284b-475c-8f28-d8b320be94b3" containerName="istio-proxy"
Nov 03 09:24:33.189969 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.189824    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb6f0603-0fa2-4223-b0d6-5411752e25cc" containerName="istio-proxy"
Nov 03 09:24:33.189969 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.189840    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d33af3b-997c-4af3-8e58-fa2b1b78135b" containerName="istio-proxy"
Nov 03 09:24:33.189969 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.189857    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7892f9a-7f14-4b08-b5a9-933572249e05" containerName="istio-proxy"
Nov 03 09:24:33.189969 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.189866    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="03b06f04-3ec5-4d68-847c-1eb86a519e5f" containerName="istio-proxy"
Nov 03 09:24:33.195435 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.195415    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-f9b517/auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w"
Nov 03 09:24:33.247603 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.247570    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-f9b517/auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w"]
Nov 03 09:24:33.294331 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.294298    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-f9b517\"/\"openshift-service-ca.crt\""
Nov 03 09:24:33.294331 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.294315    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-f9b517\"/\"auth-test-model-f9b517-kserve-self-signed-certs\""
Nov 03 09:24:33.294551 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.294374    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-f9b517\"/\"default-dockercfg-2s2sn\""
Nov 03 09:24:33.299223 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.299202    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-f9b517\"/\"kube-root-ca.crt\""
Nov 03 09:24:33.340416 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.336436    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/254e50ff-0d1c-48f6-83ca-a88d9effba4d-dshm\") pod \"auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w\" (UID: \"254e50ff-0d1c-48f6-83ca-a88d9effba4d\") " pod="ai-tenant-e2e-shared-a-w0-f9b517/auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w"
Nov 03 09:24:33.340416 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.336498    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/254e50ff-0d1c-48f6-83ca-a88d9effba4d-model-cache\") pod \"auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w\" (UID: \"254e50ff-0d1c-48f6-83ca-a88d9effba4d\") " pod="ai-tenant-e2e-shared-a-w0-f9b517/auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w"
Nov 03 09:24:33.340416 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.336568    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/254e50ff-0d1c-48f6-83ca-a88d9effba4d-tmp-dir\") pod \"auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w\" (UID: \"254e50ff-0d1c-48f6-83ca-a88d9effba4d\") " pod="ai-tenant-e2e-shared-a-w0-f9b517/auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w"
Nov 03 09:24:33.340416 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.336605    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbjvk\" (UniqueName: \"kubernetes.io/projected/254e50ff-0d1c-48f6-83ca-a88d9effba4d-kube-api-access-vbjvk\") pod \"auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w\" (UID: \"254e50ff-0d1c-48f6-83ca-a88d9effba4d\") " pod="ai-tenant-e2e-shared-a-w0-f9b517/auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w"
Nov 03 09:24:33.340416 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.336666    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/254e50ff-0d1c-48f6-83ca-a88d9effba4d-home\") pod \"auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w\" (UID: \"254e50ff-0d1c-48f6-83ca-a88d9effba4d\") " pod="ai-tenant-e2e-shared-a-w0-f9b517/auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w"
Nov 03 09:24:33.340416 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.336712    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/254e50ff-0d1c-48f6-83ca-a88d9effba4d-tls-certs\") pod \"auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w\" (UID: \"254e50ff-0d1c-48f6-83ca-a88d9effba4d\") " pod="ai-tenant-e2e-shared-a-w0-f9b517/auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w"
Nov 03 09:24:33.438529 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.438433    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/254e50ff-0d1c-48f6-83ca-a88d9effba4d-tmp-dir\") pod \"auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w\" (UID: \"254e50ff-0d1c-48f6-83ca-a88d9effba4d\") " pod="ai-tenant-e2e-shared-a-w0-f9b517/auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w"
Nov 03 09:24:33.438529 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.438481    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbjvk\" (UniqueName: \"kubernetes.io/projected/254e50ff-0d1c-48f6-83ca-a88d9effba4d-kube-api-access-vbjvk\") pod \"auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w\" (UID: \"254e50ff-0d1c-48f6-83ca-a88d9effba4d\") " pod="ai-tenant-e2e-shared-a-w0-f9b517/auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w"
Nov 03 09:24:33.438773 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.438534    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/254e50ff-0d1c-48f6-83ca-a88d9effba4d-home\") pod \"auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w\" (UID: \"254e50ff-0d1c-48f6-83ca-a88d9effba4d\") " pod="ai-tenant-e2e-shared-a-w0-f9b517/auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w"
Nov 03 09:24:33.438773 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.438564    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/254e50ff-0d1c-48f6-83ca-a88d9effba4d-tls-certs\") pod \"auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w\" (UID: \"254e50ff-0d1c-48f6-83ca-a88d9effba4d\") " pod="ai-tenant-e2e-shared-a-w0-f9b517/auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w"
Nov 03 09:24:33.438773 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.438699    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/254e50ff-0d1c-48f6-83ca-a88d9effba4d-dshm\") pod \"auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w\" (UID: \"254e50ff-0d1c-48f6-83ca-a88d9effba4d\") " pod="ai-tenant-e2e-shared-a-w0-f9b517/auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w"
Nov 03 09:24:33.438773 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.438740    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/254e50ff-0d1c-48f6-83ca-a88d9effba4d-model-cache\") pod \"auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w\" (UID: \"254e50ff-0d1c-48f6-83ca-a88d9effba4d\") " pod="ai-tenant-e2e-shared-a-w0-f9b517/auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w"
Nov 03 09:24:33.439040 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.438878    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/254e50ff-0d1c-48f6-83ca-a88d9effba4d-tmp-dir\") pod \"auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w\" (UID: \"254e50ff-0d1c-48f6-83ca-a88d9effba4d\") " pod="ai-tenant-e2e-shared-a-w0-f9b517/auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w"
Nov 03 09:24:33.439040 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.438903    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/254e50ff-0d1c-48f6-83ca-a88d9effba4d-home\") pod \"auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w\" (UID: \"254e50ff-0d1c-48f6-83ca-a88d9effba4d\") " pod="ai-tenant-e2e-shared-a-w0-f9b517/auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w"
Nov 03 09:24:33.439161 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.439124    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/254e50ff-0d1c-48f6-83ca-a88d9effba4d-model-cache\") pod \"auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w\" (UID: \"254e50ff-0d1c-48f6-83ca-a88d9effba4d\") " pod="ai-tenant-e2e-shared-a-w0-f9b517/auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w"
Nov 03 09:24:33.441544 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.441524    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/254e50ff-0d1c-48f6-83ca-a88d9effba4d-dshm\") pod \"auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w\" (UID: \"254e50ff-0d1c-48f6-83ca-a88d9effba4d\") " pod="ai-tenant-e2e-shared-a-w0-f9b517/auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w"
Nov 03 09:24:33.441655 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.441553    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/254e50ff-0d1c-48f6-83ca-a88d9effba4d-tls-certs\") pod \"auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w\" (UID: \"254e50ff-0d1c-48f6-83ca-a88d9effba4d\") " pod="ai-tenant-e2e-shared-a-w0-f9b517/auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w"
Nov 03 09:24:33.498271 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.498238    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbjvk\" (UniqueName: \"kubernetes.io/projected/254e50ff-0d1c-48f6-83ca-a88d9effba4d-kube-api-access-vbjvk\") pod \"auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w\" (UID: \"254e50ff-0d1c-48f6-83ca-a88d9effba4d\") " pod="ai-tenant-e2e-shared-a-w0-f9b517/auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w"
Nov 03 09:24:33.508191 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.508153    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-f9b517/auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w"
Nov 03 09:24:33.900549 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:33.900519    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-f9b517/auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w"]
Nov 03 09:24:33.902465 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:24:33.902438    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod254e50ff_0d1c_48f6_83ca_a88d9effba4d.slice/crio-40ab3f4a94e6a2e8a06a25edb512e0da3dc7250ddba3322f3f27dadb79eec469 WatchSource:0}: Error finding container 40ab3f4a94e6a2e8a06a25edb512e0da3dc7250ddba3322f3f27dadb79eec469: Status 404 returned error can't find the container with id 40ab3f4a94e6a2e8a06a25edb512e0da3dc7250ddba3322f3f27dadb79eec469
Nov 03 09:24:34.912517 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:34.912479    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-f9b517/auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w" event={"ID":"254e50ff-0d1c-48f6-83ca-a88d9effba4d","Type":"ContainerStarted","Data":"ad2d583e863e7bf92a92c49dda008703032904d4e0a0373c6e89368cf0a64afe"}
Nov 03 09:24:34.912976 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:34.912527    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-f9b517/auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w" event={"ID":"254e50ff-0d1c-48f6-83ca-a88d9effba4d","Type":"ContainerStarted","Data":"40ab3f4a94e6a2e8a06a25edb512e0da3dc7250ddba3322f3f27dadb79eec469"}
Nov 03 09:24:35.031424 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:35.031365    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-f9b517/auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w" podStartSLOduration=2.031351072 podStartE2EDuration="2.031351072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:24:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:24:35.028720648 +0000 UTC m=+2699.297784115" watchObservedRunningTime="2026-11-03 09:24:35.031351072 +0000 UTC m=+2699.300414548"
Nov 03 09:24:35.509208 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:35.509175    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-f9b517/auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w"
Nov 03 09:24:36.399617 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:36.399590    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 09:24:36.403027 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:36.403005    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 09:24:37.987915 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:37.987885    4473 scope.go:122] "RemoveContainer" containerID="dfbbb06c23abd7b62ea93ad06f2304b0544743612e2249fa7ad8287f6438725f"
Nov 03 09:24:37.997528 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:37.997506    4473 scope.go:122] "RemoveContainer" containerID="ddf8ae3056e9f37019bfa451a7f7d5a929e145ef4e7113f94ec1f372f7b44b54"
Nov 03 09:24:38.006657 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:38.006638    4473 scope.go:122] "RemoveContainer" containerID="309c6dc8e1062ae5edbbfdf0c6d0c3cf6e087712079c15d99328851c8c992d15"
Nov 03 09:24:38.015059 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:38.015038    4473 scope.go:122] "RemoveContainer" containerID="3290c3fcc4d84430a2aad5d4599b547a624cab6c61b4e3a780e8ceb25f98785a"
Nov 03 09:24:43.508894 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:43.508859    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-f9b517/auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w"
Nov 03 09:24:43.521315 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:43.521293    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-f9b517/auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w"
Nov 03 09:24:43.981734 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:43.981707    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-f9b517/auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w"
Nov 03 09:24:46.937500 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:24:46.937464    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:24:55.814755 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:24:55.814723    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:25:13.363681 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:13.363643    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-b196c8/auth-test-model-b196c8-kserve-588d68f749-4t8w7"]
Nov 03 09:25:13.367300 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:13.367278    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b196c8/auth-test-model-b196c8-kserve-588d68f749-4t8w7"
Nov 03 09:25:13.402786 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:13.402752    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-home\") pod \"auth-test-model-b196c8-kserve-588d68f749-4t8w7\" (UID: \"c7ce6825-d6c3-4d53-8759-0c0fb34b1f46\") " pod="ai-tenant-e2e-shared-b-w3-b196c8/auth-test-model-b196c8-kserve-588d68f749-4t8w7"
Nov 03 09:25:13.402979 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:13.402806    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-dshm\") pod \"auth-test-model-b196c8-kserve-588d68f749-4t8w7\" (UID: \"c7ce6825-d6c3-4d53-8759-0c0fb34b1f46\") " pod="ai-tenant-e2e-shared-b-w3-b196c8/auth-test-model-b196c8-kserve-588d68f749-4t8w7"
Nov 03 09:25:13.403071 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:13.403011    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-model-cache\") pod \"auth-test-model-b196c8-kserve-588d68f749-4t8w7\" (UID: \"c7ce6825-d6c3-4d53-8759-0c0fb34b1f46\") " pod="ai-tenant-e2e-shared-b-w3-b196c8/auth-test-model-b196c8-kserve-588d68f749-4t8w7"
Nov 03 09:25:13.403134 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:13.403097    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-tls-certs\") pod \"auth-test-model-b196c8-kserve-588d68f749-4t8w7\" (UID: \"c7ce6825-d6c3-4d53-8759-0c0fb34b1f46\") " pod="ai-tenant-e2e-shared-b-w3-b196c8/auth-test-model-b196c8-kserve-588d68f749-4t8w7"
Nov 03 09:25:13.403134 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:13.403126    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-tmp-dir\") pod \"auth-test-model-b196c8-kserve-588d68f749-4t8w7\" (UID: \"c7ce6825-d6c3-4d53-8759-0c0fb34b1f46\") " pod="ai-tenant-e2e-shared-b-w3-b196c8/auth-test-model-b196c8-kserve-588d68f749-4t8w7"
Nov 03 09:25:13.403266 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:13.403248    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbkld\" (UniqueName: \"kubernetes.io/projected/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-kube-api-access-bbkld\") pod \"auth-test-model-b196c8-kserve-588d68f749-4t8w7\" (UID: \"c7ce6825-d6c3-4d53-8759-0c0fb34b1f46\") " pod="ai-tenant-e2e-shared-b-w3-b196c8/auth-test-model-b196c8-kserve-588d68f749-4t8w7"
Nov 03 09:25:13.438833 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:13.438799    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b196c8\"/\"openshift-service-ca.crt\""
Nov 03 09:25:13.443312 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:13.443284    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b196c8/auth-test-model-b196c8-kserve-588d68f749-4t8w7"]
Nov 03 09:25:13.472114 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:13.472088    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b196c8\"/\"auth-test-model-b196c8-kserve-self-signed-certs\""
Nov 03 09:25:13.472243 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:13.472116    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b196c8\"/\"kube-root-ca.crt\""
Nov 03 09:25:13.472243 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:13.472163    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b196c8\"/\"default-dockercfg-99khw\""
Nov 03 09:25:13.504322 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:13.504291    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-home\") pod \"auth-test-model-b196c8-kserve-588d68f749-4t8w7\" (UID: \"c7ce6825-d6c3-4d53-8759-0c0fb34b1f46\") " pod="ai-tenant-e2e-shared-b-w3-b196c8/auth-test-model-b196c8-kserve-588d68f749-4t8w7"
Nov 03 09:25:13.505041 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:13.505017    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-dshm\") pod \"auth-test-model-b196c8-kserve-588d68f749-4t8w7\" (UID: \"c7ce6825-d6c3-4d53-8759-0c0fb34b1f46\") " pod="ai-tenant-e2e-shared-b-w3-b196c8/auth-test-model-b196c8-kserve-588d68f749-4t8w7"
Nov 03 09:25:13.505403 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:13.505375    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-model-cache\") pod \"auth-test-model-b196c8-kserve-588d68f749-4t8w7\" (UID: \"c7ce6825-d6c3-4d53-8759-0c0fb34b1f46\") " pod="ai-tenant-e2e-shared-b-w3-b196c8/auth-test-model-b196c8-kserve-588d68f749-4t8w7"
Nov 03 09:25:13.505492 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:13.505444    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-tls-certs\") pod \"auth-test-model-b196c8-kserve-588d68f749-4t8w7\" (UID: \"c7ce6825-d6c3-4d53-8759-0c0fb34b1f46\") " pod="ai-tenant-e2e-shared-b-w3-b196c8/auth-test-model-b196c8-kserve-588d68f749-4t8w7"
Nov 03 09:25:13.505492 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:13.505470    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-tmp-dir\") pod \"auth-test-model-b196c8-kserve-588d68f749-4t8w7\" (UID: \"c7ce6825-d6c3-4d53-8759-0c0fb34b1f46\") " pod="ai-tenant-e2e-shared-b-w3-b196c8/auth-test-model-b196c8-kserve-588d68f749-4t8w7"
Nov 03 09:25:13.505608 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:13.505588    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbkld\" (UniqueName: \"kubernetes.io/projected/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-kube-api-access-bbkld\") pod \"auth-test-model-b196c8-kserve-588d68f749-4t8w7\" (UID: \"c7ce6825-d6c3-4d53-8759-0c0fb34b1f46\") " pod="ai-tenant-e2e-shared-b-w3-b196c8/auth-test-model-b196c8-kserve-588d68f749-4t8w7"
Nov 03 09:25:13.506079 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:13.506056    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-model-cache\") pod \"auth-test-model-b196c8-kserve-588d68f749-4t8w7\" (UID: \"c7ce6825-d6c3-4d53-8759-0c0fb34b1f46\") " pod="ai-tenant-e2e-shared-b-w3-b196c8/auth-test-model-b196c8-kserve-588d68f749-4t8w7"
Nov 03 09:25:13.506236 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:13.504884    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-home\") pod \"auth-test-model-b196c8-kserve-588d68f749-4t8w7\" (UID: \"c7ce6825-d6c3-4d53-8759-0c0fb34b1f46\") " pod="ai-tenant-e2e-shared-b-w3-b196c8/auth-test-model-b196c8-kserve-588d68f749-4t8w7"
Nov 03 09:25:13.506563 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:13.506524    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-tmp-dir\") pod \"auth-test-model-b196c8-kserve-588d68f749-4t8w7\" (UID: \"c7ce6825-d6c3-4d53-8759-0c0fb34b1f46\") " pod="ai-tenant-e2e-shared-b-w3-b196c8/auth-test-model-b196c8-kserve-588d68f749-4t8w7"
Nov 03 09:25:13.509563 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:13.509515    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-dshm\") pod \"auth-test-model-b196c8-kserve-588d68f749-4t8w7\" (UID: \"c7ce6825-d6c3-4d53-8759-0c0fb34b1f46\") " pod="ai-tenant-e2e-shared-b-w3-b196c8/auth-test-model-b196c8-kserve-588d68f749-4t8w7"
Nov 03 09:25:13.523312 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:13.522314    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-tls-certs\") pod \"auth-test-model-b196c8-kserve-588d68f749-4t8w7\" (UID: \"c7ce6825-d6c3-4d53-8759-0c0fb34b1f46\") " pod="ai-tenant-e2e-shared-b-w3-b196c8/auth-test-model-b196c8-kserve-588d68f749-4t8w7"
Nov 03 09:25:13.565043 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:13.565009    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbkld\" (UniqueName: \"kubernetes.io/projected/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-kube-api-access-bbkld\") pod \"auth-test-model-b196c8-kserve-588d68f749-4t8w7\" (UID: \"c7ce6825-d6c3-4d53-8759-0c0fb34b1f46\") " pod="ai-tenant-e2e-shared-b-w3-b196c8/auth-test-model-b196c8-kserve-588d68f749-4t8w7"
Nov 03 09:25:13.679474 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:13.679431    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b196c8/auth-test-model-b196c8-kserve-588d68f749-4t8w7"
Nov 03 09:25:14.076287 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:14.076262    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b196c8/auth-test-model-b196c8-kserve-588d68f749-4t8w7"]
Nov 03 09:25:14.077482 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:25:14.077459    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7ce6825_d6c3_4d53_8759_0c0fb34b1f46.slice/crio-f01731cd5513a1c387dfb30db4d988d949df4a08e21619ed2cc5cb7efa7ed30d WatchSource:0}: Error finding container f01731cd5513a1c387dfb30db4d988d949df4a08e21619ed2cc5cb7efa7ed30d: Status 404 returned error can't find the container with id f01731cd5513a1c387dfb30db4d988d949df4a08e21619ed2cc5cb7efa7ed30d
Nov 03 09:25:14.159631 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:14.159595    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b196c8/auth-test-model-b196c8-kserve-588d68f749-4t8w7" event={"ID":"c7ce6825-d6c3-4d53-8759-0c0fb34b1f46","Type":"ContainerStarted","Data":"435f3d0e9c1f2eec439cd11e69f78e4d10566507edc22645804eed51c221cba4"}
Nov 03 09:25:14.159771 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:14.159668    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b196c8/auth-test-model-b196c8-kserve-588d68f749-4t8w7" event={"ID":"c7ce6825-d6c3-4d53-8759-0c0fb34b1f46","Type":"ContainerStarted","Data":"f01731cd5513a1c387dfb30db4d988d949df4a08e21619ed2cc5cb7efa7ed30d"}
Nov 03 09:25:14.279220 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:14.279170    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-b196c8/auth-test-model-b196c8-kserve-588d68f749-4t8w7" podStartSLOduration=1.279155445 podStartE2EDuration="1.279155445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:25:14.276468373 +0000 UTC m=+2738.545531827" watchObservedRunningTime="2026-11-03 09:25:14.279155445 +0000 UTC m=+2738.548218875"
Nov 03 09:25:14.680488 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:14.680460    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-b196c8/auth-test-model-b196c8-kserve-588d68f749-4t8w7"
Nov 03 09:25:16.937838 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:25:16.937759    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:25:23.680608 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:23.680577    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-b196c8/auth-test-model-b196c8-kserve-588d68f749-4t8w7"
Nov 03 09:25:23.693926 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:23.693901    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-b196c8/auth-test-model-b196c8-kserve-588d68f749-4t8w7"
Nov 03 09:25:24.230130 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:24.230102    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-b196c8/auth-test-model-b196c8-kserve-588d68f749-4t8w7"
Nov 03 09:25:28.300850 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:28.300821    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-3df1cc/auth-test-model-3df1cc-kserve-74596bcc89-6xm2k"]
Nov 03 09:25:28.304327 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:28.304308    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-3df1cc/auth-test-model-3df1cc-kserve-74596bcc89-6xm2k"
Nov 03 09:25:28.345880 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:28.345854    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-home\") pod \"auth-test-model-3df1cc-kserve-74596bcc89-6xm2k\" (UID: \"0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e\") " pod="ai-tenant-e2e-shared-b-w1-3df1cc/auth-test-model-3df1cc-kserve-74596bcc89-6xm2k"
Nov 03 09:25:28.346056 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:28.346031    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-tls-certs\") pod \"auth-test-model-3df1cc-kserve-74596bcc89-6xm2k\" (UID: \"0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e\") " pod="ai-tenant-e2e-shared-b-w1-3df1cc/auth-test-model-3df1cc-kserve-74596bcc89-6xm2k"
Nov 03 09:25:28.346148 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:28.346127    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-model-cache\") pod \"auth-test-model-3df1cc-kserve-74596bcc89-6xm2k\" (UID: \"0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e\") " pod="ai-tenant-e2e-shared-b-w1-3df1cc/auth-test-model-3df1cc-kserve-74596bcc89-6xm2k"
Nov 03 09:25:28.346210 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:28.346198    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-dshm\") pod \"auth-test-model-3df1cc-kserve-74596bcc89-6xm2k\" (UID: \"0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e\") " pod="ai-tenant-e2e-shared-b-w1-3df1cc/auth-test-model-3df1cc-kserve-74596bcc89-6xm2k"
Nov 03 09:25:28.346280 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:28.346261    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4zf7\" (UniqueName: \"kubernetes.io/projected/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-kube-api-access-h4zf7\") pod \"auth-test-model-3df1cc-kserve-74596bcc89-6xm2k\" (UID: \"0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e\") " pod="ai-tenant-e2e-shared-b-w1-3df1cc/auth-test-model-3df1cc-kserve-74596bcc89-6xm2k"
Nov 03 09:25:28.346358 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:28.346343    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-tmp-dir\") pod \"auth-test-model-3df1cc-kserve-74596bcc89-6xm2k\" (UID: \"0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e\") " pod="ai-tenant-e2e-shared-b-w1-3df1cc/auth-test-model-3df1cc-kserve-74596bcc89-6xm2k"
Nov 03 09:25:28.359816 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:28.359795    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-3df1cc\"/\"openshift-service-ca.crt\""
Nov 03 09:25:28.410174 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:28.410145    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-3df1cc\"/\"kube-root-ca.crt\""
Nov 03 09:25:28.410410 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:28.410392    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-3df1cc\"/\"default-dockercfg-f69w8\""
Nov 03 09:25:28.410560 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:28.410546    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-3df1cc\"/\"auth-test-model-3df1cc-kserve-self-signed-certs\""
Nov 03 09:25:28.413264 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:28.413241    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-3df1cc/auth-test-model-3df1cc-kserve-74596bcc89-6xm2k"]
Nov 03 09:25:28.448148 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:28.448119    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-dshm\") pod \"auth-test-model-3df1cc-kserve-74596bcc89-6xm2k\" (UID: \"0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e\") " pod="ai-tenant-e2e-shared-b-w1-3df1cc/auth-test-model-3df1cc-kserve-74596bcc89-6xm2k"
Nov 03 09:25:28.448275 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:28.448163    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4zf7\" (UniqueName: \"kubernetes.io/projected/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-kube-api-access-h4zf7\") pod \"auth-test-model-3df1cc-kserve-74596bcc89-6xm2k\" (UID: \"0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e\") " pod="ai-tenant-e2e-shared-b-w1-3df1cc/auth-test-model-3df1cc-kserve-74596bcc89-6xm2k"
Nov 03 09:25:28.448275 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:28.448240    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-tmp-dir\") pod \"auth-test-model-3df1cc-kserve-74596bcc89-6xm2k\" (UID: \"0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e\") " pod="ai-tenant-e2e-shared-b-w1-3df1cc/auth-test-model-3df1cc-kserve-74596bcc89-6xm2k"
Nov 03 09:25:28.448383 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:28.448353    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-home\") pod \"auth-test-model-3df1cc-kserve-74596bcc89-6xm2k\" (UID: \"0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e\") " pod="ai-tenant-e2e-shared-b-w1-3df1cc/auth-test-model-3df1cc-kserve-74596bcc89-6xm2k"
Nov 03 09:25:28.448466 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:28.448437    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-tls-certs\") pod \"auth-test-model-3df1cc-kserve-74596bcc89-6xm2k\" (UID: \"0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e\") " pod="ai-tenant-e2e-shared-b-w1-3df1cc/auth-test-model-3df1cc-kserve-74596bcc89-6xm2k"
Nov 03 09:25:28.448532 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:28.448498    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-model-cache\") pod \"auth-test-model-3df1cc-kserve-74596bcc89-6xm2k\" (UID: \"0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e\") " pod="ai-tenant-e2e-shared-b-w1-3df1cc/auth-test-model-3df1cc-kserve-74596bcc89-6xm2k"
Nov 03 09:25:28.448792 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:28.448769    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-tmp-dir\") pod \"auth-test-model-3df1cc-kserve-74596bcc89-6xm2k\" (UID: \"0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e\") " pod="ai-tenant-e2e-shared-b-w1-3df1cc/auth-test-model-3df1cc-kserve-74596bcc89-6xm2k"
Nov 03 09:25:28.449133 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:28.448864    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-home\") pod \"auth-test-model-3df1cc-kserve-74596bcc89-6xm2k\" (UID: \"0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e\") " pod="ai-tenant-e2e-shared-b-w1-3df1cc/auth-test-model-3df1cc-kserve-74596bcc89-6xm2k"
Nov 03 09:25:28.449133 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:28.448974    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-model-cache\") pod \"auth-test-model-3df1cc-kserve-74596bcc89-6xm2k\" (UID: \"0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e\") " pod="ai-tenant-e2e-shared-b-w1-3df1cc/auth-test-model-3df1cc-kserve-74596bcc89-6xm2k"
Nov 03 09:25:28.451049 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:28.451024    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-dshm\") pod \"auth-test-model-3df1cc-kserve-74596bcc89-6xm2k\" (UID: \"0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e\") " pod="ai-tenant-e2e-shared-b-w1-3df1cc/auth-test-model-3df1cc-kserve-74596bcc89-6xm2k"
Nov 03 09:25:28.451568 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:28.451520    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-tls-certs\") pod \"auth-test-model-3df1cc-kserve-74596bcc89-6xm2k\" (UID: \"0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e\") " pod="ai-tenant-e2e-shared-b-w1-3df1cc/auth-test-model-3df1cc-kserve-74596bcc89-6xm2k"
Nov 03 09:25:28.509752 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:28.509726    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4zf7\" (UniqueName: \"kubernetes.io/projected/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-kube-api-access-h4zf7\") pod \"auth-test-model-3df1cc-kserve-74596bcc89-6xm2k\" (UID: \"0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e\") " pod="ai-tenant-e2e-shared-b-w1-3df1cc/auth-test-model-3df1cc-kserve-74596bcc89-6xm2k"
Nov 03 09:25:28.617075 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:28.617045    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-3df1cc/auth-test-model-3df1cc-kserve-74596bcc89-6xm2k"
Nov 03 09:25:29.006933 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:25:29.006898    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b8be3ca_23d3_4edd_94fb_41ee0a7e4d9e.slice/crio-46210343e157776b5ea6590303a01c01187d9f3824c9e1dc4468a0b8e0387a58 WatchSource:0}: Error finding container 46210343e157776b5ea6590303a01c01187d9f3824c9e1dc4468a0b8e0387a58: Status 404 returned error can't find the container with id 46210343e157776b5ea6590303a01c01187d9f3824c9e1dc4468a0b8e0387a58
Nov 03 09:25:29.008790 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:29.008769    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-3df1cc/auth-test-model-3df1cc-kserve-74596bcc89-6xm2k"]
Nov 03 09:25:29.248332 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:29.248300    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-3df1cc/auth-test-model-3df1cc-kserve-74596bcc89-6xm2k" event={"ID":"0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e","Type":"ContainerStarted","Data":"fe61a7e63f74925d4131c1ccd6919223c40b4b17582c88d41febfade36083fd0"}
Nov 03 09:25:29.248527 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:29.248344    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-3df1cc/auth-test-model-3df1cc-kserve-74596bcc89-6xm2k" event={"ID":"0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e","Type":"ContainerStarted","Data":"46210343e157776b5ea6590303a01c01187d9f3824c9e1dc4468a0b8e0387a58"}
Nov 03 09:25:29.365899 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:29.365853    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-3df1cc/auth-test-model-3df1cc-kserve-74596bcc89-6xm2k" podStartSLOduration=1.365837756 podStartE2EDuration="1.365837756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:25:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:25:29.364896866 +0000 UTC m=+2753.633960327" watchObservedRunningTime="2026-11-03 09:25:29.365837756 +0000 UTC m=+2753.634901191"
Nov 03 09:25:29.617451 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:29.617377    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-3df1cc/auth-test-model-3df1cc-kserve-74596bcc89-6xm2k"
Nov 03 09:25:34.825163 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:34.825120    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:25:38.618118 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:38.618087    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-3df1cc/auth-test-model-3df1cc-kserve-74596bcc89-6xm2k"
Nov 03 09:25:38.630349 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:38.630324    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-3df1cc/auth-test-model-3df1cc-kserve-74596bcc89-6xm2k"
Nov 03 09:25:39.316897 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:39.316865    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-3df1cc/auth-test-model-3df1cc-kserve-74596bcc89-6xm2k"
Nov 03 09:25:43.019821 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:43.019789    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"]
Nov 03 09:25:43.020283 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:43.020026    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns" podUID="6da2e774-1fbc-445e-88f2-2e08a2b3b7e3" containerName="istio-proxy" containerID="cri-o://a5abe396d6eaa91ccd0702015c9207889997f2d67883a9da10fda1fc1b82ab33" gracePeriod=30
Nov 03 09:25:46.938878 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:25:46.938833    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:25:47.206066 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:47.205991    4473 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 09:25:47.206216 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:47.206066    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns" podUID="6da2e774-1fbc-445e-88f2-2e08a2b3b7e3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 09:25:48.286686 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:48.286624    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"
Nov 03 09:25:48.360743 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:48.360711    4473 generic.go:356] "Generic (PLEG): container finished" podID="6da2e774-1fbc-445e-88f2-2e08a2b3b7e3" containerID="a5abe396d6eaa91ccd0702015c9207889997f2d67883a9da10fda1fc1b82ab33" exitCode=0
Nov 03 09:25:48.360900 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:48.360769    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"
Nov 03 09:25:48.360900 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:48.360781    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns" event={"ID":"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3","Type":"ContainerDied","Data":"a5abe396d6eaa91ccd0702015c9207889997f2d67883a9da10fda1fc1b82ab33"}
Nov 03 09:25:48.360900 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:48.360823    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns" event={"ID":"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3","Type":"ContainerDied","Data":"d52fdaaeaecefcbf3cc4fc7d312d1a536e842f5d3ffc131e9fc53f600ed1599c"}
Nov 03 09:25:48.360900 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:48.360844    4473 scope.go:122] "RemoveContainer" containerID="a5abe396d6eaa91ccd0702015c9207889997f2d67883a9da10fda1fc1b82ab33"
Nov 03 09:25:48.370441 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:48.370423    4473 scope.go:122] "RemoveContainer" containerID="a5abe396d6eaa91ccd0702015c9207889997f2d67883a9da10fda1fc1b82ab33"
Nov 03 09:25:48.370732 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:25:48.370712    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5abe396d6eaa91ccd0702015c9207889997f2d67883a9da10fda1fc1b82ab33\": container with ID starting with a5abe396d6eaa91ccd0702015c9207889997f2d67883a9da10fda1fc1b82ab33 not found: ID does not exist" containerID="a5abe396d6eaa91ccd0702015c9207889997f2d67883a9da10fda1fc1b82ab33"
Nov 03 09:25:48.370802 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:48.370744    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5abe396d6eaa91ccd0702015c9207889997f2d67883a9da10fda1fc1b82ab33"} err="failed to get container status \"a5abe396d6eaa91ccd0702015c9207889997f2d67883a9da10fda1fc1b82ab33\": rpc error: code = NotFound desc = could not find container \"a5abe396d6eaa91ccd0702015c9207889997f2d67883a9da10fda1fc1b82ab33\": container with ID starting with a5abe396d6eaa91ccd0702015c9207889997f2d67883a9da10fda1fc1b82ab33 not found: ID does not exist"
Nov 03 09:25:48.467891 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:48.467858    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-workload-certs\") pod \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\" (UID: \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\") "
Nov 03 09:25:48.468093 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:48.467900    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-workload-socket\") pod \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\" (UID: \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\") "
Nov 03 09:25:48.468093 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:48.467976    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-credential-socket\") pod \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\" (UID: \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\") "
Nov 03 09:25:48.468093 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:48.468005    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-istio-podinfo\") pod \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\" (UID: \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\") "
Nov 03 09:25:48.468093 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:48.468091    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-kube-api-access-vpf9q\" (UniqueName: \"kubernetes.io/projected/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-kube-api-access-vpf9q\") pod \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\" (UID: \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\") "
Nov 03 09:25:48.468270 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:48.468123    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-istio-token\" (UniqueName: \"kubernetes.io/projected/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-istio-token\") pod \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\" (UID: \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\") "
Nov 03 09:25:48.468270 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:48.468161    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-istiod-ca-cert\") pod \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\" (UID: \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\") "
Nov 03 09:25:48.468270 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:48.468192    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-workload-certs" pod "6da2e774-1fbc-445e-88f2-2e08a2b3b7e3" (UID: "6da2e774-1fbc-445e-88f2-2e08a2b3b7e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:25:48.468270 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:48.468210    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-istio-envoy\") pod \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\" (UID: \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\") "
Nov 03 09:25:48.468270 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:48.468263    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-istio-data\") pod \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\" (UID: \"6da2e774-1fbc-445e-88f2-2e08a2b3b7e3\") "
Nov 03 09:25:48.468502 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:48.468267    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-workload-socket" pod "6da2e774-1fbc-445e-88f2-2e08a2b3b7e3" (UID: "6da2e774-1fbc-445e-88f2-2e08a2b3b7e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:25:48.468607 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:48.468516    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-istiod-ca-cert" pod "6da2e774-1fbc-445e-88f2-2e08a2b3b7e3" (UID: "6da2e774-1fbc-445e-88f2-2e08a2b3b7e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:25:48.468772 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:48.468596    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-istio-data" pod "6da2e774-1fbc-445e-88f2-2e08a2b3b7e3" (UID: "6da2e774-1fbc-445e-88f2-2e08a2b3b7e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:25:48.468872 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:48.468621    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-credential-socket" pod "6da2e774-1fbc-445e-88f2-2e08a2b3b7e3" (UID: "6da2e774-1fbc-445e-88f2-2e08a2b3b7e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:25:48.468872 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:48.468801    4473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-workload-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:25:48.469004 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:48.468880    4473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-workload-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:25:48.471050 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:48.471020    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-istio-token" pod "6da2e774-1fbc-445e-88f2-2e08a2b3b7e3" (UID: "6da2e774-1fbc-445e-88f2-2e08a2b3b7e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:25:48.471050 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:48.471034    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-istio-envoy" pod "6da2e774-1fbc-445e-88f2-2e08a2b3b7e3" (UID: "6da2e774-1fbc-445e-88f2-2e08a2b3b7e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:25:48.471186 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:48.471062    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-istio-podinfo" pod "6da2e774-1fbc-445e-88f2-2e08a2b3b7e3" (UID: "6da2e774-1fbc-445e-88f2-2e08a2b3b7e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 09:25:48.471186 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:48.471072    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-kube-api-access-vpf9q" pod "6da2e774-1fbc-445e-88f2-2e08a2b3b7e3" (UID: "6da2e774-1fbc-445e-88f2-2e08a2b3b7e3"). InnerVolumeSpecName "kube-api-access-vpf9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:25:48.570126 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:48.570050    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpf9q\" (UniqueName: \"kubernetes.io/projected/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-kube-api-access-vpf9q\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:25:48.570126 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:48.570079    4473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-istio-token\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:25:48.570126 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:48.570088    4473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-istiod-ca-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:25:48.570126 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:48.570100    4473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-istio-envoy\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:25:48.570126 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:48.570109    4473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-istio-data\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:25:48.570126 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:48.570116    4473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-credential-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:25:48.570126 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:48.570124    4473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3-istio-podinfo\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:25:48.788394 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:48.788357    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"]
Nov 03 09:25:48.794731 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:48.794703    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-lbhns"]
Nov 03 09:25:49.884851 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:49.884803    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6da2e774-1fbc-445e-88f2-2e08a2b3b7e3" path="/var/lib/kubelet/pods/6da2e774-1fbc-445e-88f2-2e08a2b3b7e3/volumes"
Nov 03 09:25:55.932990 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:55.932959    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-2fc57e/explicit-model-2fc57e-kserve-6ff449c699-sq9dz"]
Nov 03 09:25:55.933801 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:55.933449    4473 cpu_manager.go:414] "containerMap: removing container" podUID="6da2e774-1fbc-445e-88f2-2e08a2b3b7e3" containerName="istio-proxy"
Nov 03 09:25:55.933801 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:55.933461    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6da2e774-1fbc-445e-88f2-2e08a2b3b7e3" containerName="istio-proxy"
Nov 03 09:25:55.933801 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:55.933555    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="6da2e774-1fbc-445e-88f2-2e08a2b3b7e3" containerName="istio-proxy"
Nov 03 09:25:55.936874 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:55.936858    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-2fc57e/explicit-model-2fc57e-kserve-6ff449c699-sq9dz"
Nov 03 09:25:56.042453 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:56.042408    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-2fc57e\"/\"openshift-service-ca.crt\""
Nov 03 09:25:56.042453 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:56.042434    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-2fc57e\"/\"kube-root-ca.crt\""
Nov 03 09:25:56.042453 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:56.042460    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-2fc57e\"/\"explicit-model-2fc57e-kserve-self-signed-certs\""
Nov 03 09:25:56.042697 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:56.042515    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-2fc57e\"/\"default-dockercfg-brtq2\""
Nov 03 09:25:56.045138 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:56.045120    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-2fc57e/explicit-model-2fc57e-kserve-6ff449c699-sq9dz"]
Nov 03 09:25:56.048121 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:56.048095    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-tls-certs\") pod \"explicit-model-2fc57e-kserve-6ff449c699-sq9dz\" (UID: \"60a13de7-5fc4-48d8-a1b6-90ac67bdb782\") " pod="ai-tenant-e2e-explicit-2fc57e/explicit-model-2fc57e-kserve-6ff449c699-sq9dz"
Nov 03 09:25:56.048228 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:56.048163    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-dshm\") pod \"explicit-model-2fc57e-kserve-6ff449c699-sq9dz\" (UID: \"60a13de7-5fc4-48d8-a1b6-90ac67bdb782\") " pod="ai-tenant-e2e-explicit-2fc57e/explicit-model-2fc57e-kserve-6ff449c699-sq9dz"
Nov 03 09:25:56.048275 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:56.048253    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbhq8\" (UniqueName: \"kubernetes.io/projected/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-kube-api-access-pbhq8\") pod \"explicit-model-2fc57e-kserve-6ff449c699-sq9dz\" (UID: \"60a13de7-5fc4-48d8-a1b6-90ac67bdb782\") " pod="ai-tenant-e2e-explicit-2fc57e/explicit-model-2fc57e-kserve-6ff449c699-sq9dz"
Nov 03 09:25:56.048327 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:56.048313    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-home\") pod \"explicit-model-2fc57e-kserve-6ff449c699-sq9dz\" (UID: \"60a13de7-5fc4-48d8-a1b6-90ac67bdb782\") " pod="ai-tenant-e2e-explicit-2fc57e/explicit-model-2fc57e-kserve-6ff449c699-sq9dz"
Nov 03 09:25:56.048400 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:56.048383    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-tmp-dir\") pod \"explicit-model-2fc57e-kserve-6ff449c699-sq9dz\" (UID: \"60a13de7-5fc4-48d8-a1b6-90ac67bdb782\") " pod="ai-tenant-e2e-explicit-2fc57e/explicit-model-2fc57e-kserve-6ff449c699-sq9dz"
Nov 03 09:25:56.048474 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:56.048461    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-model-cache\") pod \"explicit-model-2fc57e-kserve-6ff449c699-sq9dz\" (UID: \"60a13de7-5fc4-48d8-a1b6-90ac67bdb782\") " pod="ai-tenant-e2e-explicit-2fc57e/explicit-model-2fc57e-kserve-6ff449c699-sq9dz"
Nov 03 09:25:56.149811 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:56.149780    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-model-cache\") pod \"explicit-model-2fc57e-kserve-6ff449c699-sq9dz\" (UID: \"60a13de7-5fc4-48d8-a1b6-90ac67bdb782\") " pod="ai-tenant-e2e-explicit-2fc57e/explicit-model-2fc57e-kserve-6ff449c699-sq9dz"
Nov 03 09:25:56.150038 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:56.149853    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-tls-certs\") pod \"explicit-model-2fc57e-kserve-6ff449c699-sq9dz\" (UID: \"60a13de7-5fc4-48d8-a1b6-90ac67bdb782\") " pod="ai-tenant-e2e-explicit-2fc57e/explicit-model-2fc57e-kserve-6ff449c699-sq9dz"
Nov 03 09:25:56.150038 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:56.149912    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-dshm\") pod \"explicit-model-2fc57e-kserve-6ff449c699-sq9dz\" (UID: \"60a13de7-5fc4-48d8-a1b6-90ac67bdb782\") " pod="ai-tenant-e2e-explicit-2fc57e/explicit-model-2fc57e-kserve-6ff449c699-sq9dz"
Nov 03 09:25:56.150038 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:56.149990    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbhq8\" (UniqueName: \"kubernetes.io/projected/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-kube-api-access-pbhq8\") pod \"explicit-model-2fc57e-kserve-6ff449c699-sq9dz\" (UID: \"60a13de7-5fc4-48d8-a1b6-90ac67bdb782\") " pod="ai-tenant-e2e-explicit-2fc57e/explicit-model-2fc57e-kserve-6ff449c699-sq9dz"
Nov 03 09:25:56.150224 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:56.150092    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-home\") pod \"explicit-model-2fc57e-kserve-6ff449c699-sq9dz\" (UID: \"60a13de7-5fc4-48d8-a1b6-90ac67bdb782\") " pod="ai-tenant-e2e-explicit-2fc57e/explicit-model-2fc57e-kserve-6ff449c699-sq9dz"
Nov 03 09:25:56.150224 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:56.150124    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-tmp-dir\") pod \"explicit-model-2fc57e-kserve-6ff449c699-sq9dz\" (UID: \"60a13de7-5fc4-48d8-a1b6-90ac67bdb782\") " pod="ai-tenant-e2e-explicit-2fc57e/explicit-model-2fc57e-kserve-6ff449c699-sq9dz"
Nov 03 09:25:56.150224 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:56.150199    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-model-cache\") pod \"explicit-model-2fc57e-kserve-6ff449c699-sq9dz\" (UID: \"60a13de7-5fc4-48d8-a1b6-90ac67bdb782\") " pod="ai-tenant-e2e-explicit-2fc57e/explicit-model-2fc57e-kserve-6ff449c699-sq9dz"
Nov 03 09:25:56.150398 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:56.150385    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-home\") pod \"explicit-model-2fc57e-kserve-6ff449c699-sq9dz\" (UID: \"60a13de7-5fc4-48d8-a1b6-90ac67bdb782\") " pod="ai-tenant-e2e-explicit-2fc57e/explicit-model-2fc57e-kserve-6ff449c699-sq9dz"
Nov 03 09:25:56.150485 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:56.150465    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-tmp-dir\") pod \"explicit-model-2fc57e-kserve-6ff449c699-sq9dz\" (UID: \"60a13de7-5fc4-48d8-a1b6-90ac67bdb782\") " pod="ai-tenant-e2e-explicit-2fc57e/explicit-model-2fc57e-kserve-6ff449c699-sq9dz"
Nov 03 09:25:56.152377 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:56.152351    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-dshm\") pod \"explicit-model-2fc57e-kserve-6ff449c699-sq9dz\" (UID: \"60a13de7-5fc4-48d8-a1b6-90ac67bdb782\") " pod="ai-tenant-e2e-explicit-2fc57e/explicit-model-2fc57e-kserve-6ff449c699-sq9dz"
Nov 03 09:25:56.152514 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:56.152498    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-tls-certs\") pod \"explicit-model-2fc57e-kserve-6ff449c699-sq9dz\" (UID: \"60a13de7-5fc4-48d8-a1b6-90ac67bdb782\") " pod="ai-tenant-e2e-explicit-2fc57e/explicit-model-2fc57e-kserve-6ff449c699-sq9dz"
Nov 03 09:25:56.209562 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:56.209498    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbhq8\" (UniqueName: \"kubernetes.io/projected/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-kube-api-access-pbhq8\") pod \"explicit-model-2fc57e-kserve-6ff449c699-sq9dz\" (UID: \"60a13de7-5fc4-48d8-a1b6-90ac67bdb782\") " pod="ai-tenant-e2e-explicit-2fc57e/explicit-model-2fc57e-kserve-6ff449c699-sq9dz"
Nov 03 09:25:56.249438 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:56.249410    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-2fc57e/explicit-model-2fc57e-kserve-6ff449c699-sq9dz"
Nov 03 09:25:56.635201 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:56.635175    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-2fc57e/explicit-model-2fc57e-kserve-6ff449c699-sq9dz"]
Nov 03 09:25:56.636026 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:25:56.635993    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60a13de7_5fc4_48d8_a1b6_90ac67bdb782.slice/crio-bdcaa5da13511741899372c4999219eac935d4aae35788e6ebb84467e9447453 WatchSource:0}: Error finding container bdcaa5da13511741899372c4999219eac935d4aae35788e6ebb84467e9447453: Status 404 returned error can't find the container with id bdcaa5da13511741899372c4999219eac935d4aae35788e6ebb84467e9447453
Nov 03 09:25:57.447816 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:57.447779    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-2fc57e/explicit-model-2fc57e-kserve-6ff449c699-sq9dz" event={"ID":"60a13de7-5fc4-48d8-a1b6-90ac67bdb782","Type":"ContainerStarted","Data":"34479bd04e050d05afc8578f095d2e0d6914e05d82f0c3a41db1c1d3d9372340"}
Nov 03 09:25:57.448202 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:57.447826    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-2fc57e/explicit-model-2fc57e-kserve-6ff449c699-sq9dz" event={"ID":"60a13de7-5fc4-48d8-a1b6-90ac67bdb782","Type":"ContainerStarted","Data":"bdcaa5da13511741899372c4999219eac935d4aae35788e6ebb84467e9447453"}
Nov 03 09:25:57.580722 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:57.580660    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-2fc57e/explicit-model-2fc57e-kserve-6ff449c699-sq9dz" podStartSLOduration=2.5806462100000003 podStartE2EDuration="2.58064621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:25:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:25:57.579038192 +0000 UTC m=+2781.848101647" watchObservedRunningTime="2026-11-03 09:25:57.58064621 +0000 UTC m=+2781.849709668"
Nov 03 09:25:58.250113 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:25:58.250081    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-2fc57e/explicit-model-2fc57e-kserve-6ff449c699-sq9dz"
Nov 03 09:26:06.250375 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:06.250344    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-2fc57e/explicit-model-2fc57e-kserve-6ff449c699-sq9dz"
Nov 03 09:26:06.263465 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:06.263437    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-2fc57e/explicit-model-2fc57e-kserve-6ff449c699-sq9dz"
Nov 03 09:26:06.510530 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:06.510444    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-2fc57e/explicit-model-2fc57e-kserve-6ff449c699-sq9dz"
Nov 03 09:26:16.939252 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:26:16.939212    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:26:17.419262 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:17.419229    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:26:23.708459 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:23.708426    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-9ed110/auth-test-model-9ed110-kserve-76968d48b9-wlwcm"]
Nov 03 09:26:23.712231 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:23.712216    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9ed110/auth-test-model-9ed110-kserve-76968d48b9-wlwcm"
Nov 03 09:26:23.767817 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:23.767786    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-9ed110\"/\"default-dockercfg-5g5dj\""
Nov 03 09:26:23.792159 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:23.792128    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dc0565c-837f-432a-bf4e-3b0f611a58fa-home\") pod \"auth-test-model-9ed110-kserve-76968d48b9-wlwcm\" (UID: \"9dc0565c-837f-432a-bf4e-3b0f611a58fa\") " pod="ai-tenant-e2e-shared-b-w0-9ed110/auth-test-model-9ed110-kserve-76968d48b9-wlwcm"
Nov 03 09:26:23.792377 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:23.792350    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dc0565c-837f-432a-bf4e-3b0f611a58fa-tls-certs\") pod \"auth-test-model-9ed110-kserve-76968d48b9-wlwcm\" (UID: \"9dc0565c-837f-432a-bf4e-3b0f611a58fa\") " pod="ai-tenant-e2e-shared-b-w0-9ed110/auth-test-model-9ed110-kserve-76968d48b9-wlwcm"
Nov 03 09:26:23.792441 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:23.792407    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dc0565c-837f-432a-bf4e-3b0f611a58fa-model-cache\") pod \"auth-test-model-9ed110-kserve-76968d48b9-wlwcm\" (UID: \"9dc0565c-837f-432a-bf4e-3b0f611a58fa\") " pod="ai-tenant-e2e-shared-b-w0-9ed110/auth-test-model-9ed110-kserve-76968d48b9-wlwcm"
Nov 03 09:26:23.792501 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:23.792485    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppzdl\" (UniqueName: \"kubernetes.io/projected/9dc0565c-837f-432a-bf4e-3b0f611a58fa-kube-api-access-ppzdl\") pod \"auth-test-model-9ed110-kserve-76968d48b9-wlwcm\" (UID: \"9dc0565c-837f-432a-bf4e-3b0f611a58fa\") " pod="ai-tenant-e2e-shared-b-w0-9ed110/auth-test-model-9ed110-kserve-76968d48b9-wlwcm"
Nov 03 09:26:23.792576 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:23.792559    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dc0565c-837f-432a-bf4e-3b0f611a58fa-dshm\") pod \"auth-test-model-9ed110-kserve-76968d48b9-wlwcm\" (UID: \"9dc0565c-837f-432a-bf4e-3b0f611a58fa\") " pod="ai-tenant-e2e-shared-b-w0-9ed110/auth-test-model-9ed110-kserve-76968d48b9-wlwcm"
Nov 03 09:26:23.792659 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:23.792641    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dc0565c-837f-432a-bf4e-3b0f611a58fa-tmp-dir\") pod \"auth-test-model-9ed110-kserve-76968d48b9-wlwcm\" (UID: \"9dc0565c-837f-432a-bf4e-3b0f611a58fa\") " pod="ai-tenant-e2e-shared-b-w0-9ed110/auth-test-model-9ed110-kserve-76968d48b9-wlwcm"
Nov 03 09:26:23.818204 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:23.818178    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-9ed110\"/\"openshift-service-ca.crt\""
Nov 03 09:26:23.818204 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:23.818178    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-9ed110\"/\"auth-test-model-9ed110-kserve-self-signed-certs\""
Nov 03 09:26:23.819581 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:23.819566    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-9ed110\"/\"kube-root-ca.crt\""
Nov 03 09:26:23.821069 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:23.821050    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9ed110/auth-test-model-9ed110-kserve-76968d48b9-wlwcm"]
Nov 03 09:26:23.894283 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:23.894253    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dc0565c-837f-432a-bf4e-3b0f611a58fa-tls-certs\") pod \"auth-test-model-9ed110-kserve-76968d48b9-wlwcm\" (UID: \"9dc0565c-837f-432a-bf4e-3b0f611a58fa\") " pod="ai-tenant-e2e-shared-b-w0-9ed110/auth-test-model-9ed110-kserve-76968d48b9-wlwcm"
Nov 03 09:26:23.894467 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:23.894299    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dc0565c-837f-432a-bf4e-3b0f611a58fa-model-cache\") pod \"auth-test-model-9ed110-kserve-76968d48b9-wlwcm\" (UID: \"9dc0565c-837f-432a-bf4e-3b0f611a58fa\") " pod="ai-tenant-e2e-shared-b-w0-9ed110/auth-test-model-9ed110-kserve-76968d48b9-wlwcm"
Nov 03 09:26:23.894467 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:23.894350    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppzdl\" (UniqueName: \"kubernetes.io/projected/9dc0565c-837f-432a-bf4e-3b0f611a58fa-kube-api-access-ppzdl\") pod \"auth-test-model-9ed110-kserve-76968d48b9-wlwcm\" (UID: \"9dc0565c-837f-432a-bf4e-3b0f611a58fa\") " pod="ai-tenant-e2e-shared-b-w0-9ed110/auth-test-model-9ed110-kserve-76968d48b9-wlwcm"
Nov 03 09:26:23.894467 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:23.894451    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dc0565c-837f-432a-bf4e-3b0f611a58fa-dshm\") pod \"auth-test-model-9ed110-kserve-76968d48b9-wlwcm\" (UID: \"9dc0565c-837f-432a-bf4e-3b0f611a58fa\") " pod="ai-tenant-e2e-shared-b-w0-9ed110/auth-test-model-9ed110-kserve-76968d48b9-wlwcm"
Nov 03 09:26:23.894643 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:23.894488    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dc0565c-837f-432a-bf4e-3b0f611a58fa-tmp-dir\") pod \"auth-test-model-9ed110-kserve-76968d48b9-wlwcm\" (UID: \"9dc0565c-837f-432a-bf4e-3b0f611a58fa\") " pod="ai-tenant-e2e-shared-b-w0-9ed110/auth-test-model-9ed110-kserve-76968d48b9-wlwcm"
Nov 03 09:26:23.894643 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:23.894524    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dc0565c-837f-432a-bf4e-3b0f611a58fa-home\") pod \"auth-test-model-9ed110-kserve-76968d48b9-wlwcm\" (UID: \"9dc0565c-837f-432a-bf4e-3b0f611a58fa\") " pod="ai-tenant-e2e-shared-b-w0-9ed110/auth-test-model-9ed110-kserve-76968d48b9-wlwcm"
Nov 03 09:26:23.894856 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:23.894825    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dc0565c-837f-432a-bf4e-3b0f611a58fa-model-cache\") pod \"auth-test-model-9ed110-kserve-76968d48b9-wlwcm\" (UID: \"9dc0565c-837f-432a-bf4e-3b0f611a58fa\") " pod="ai-tenant-e2e-shared-b-w0-9ed110/auth-test-model-9ed110-kserve-76968d48b9-wlwcm"
Nov 03 09:26:23.895011 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:23.894879    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dc0565c-837f-432a-bf4e-3b0f611a58fa-tmp-dir\") pod \"auth-test-model-9ed110-kserve-76968d48b9-wlwcm\" (UID: \"9dc0565c-837f-432a-bf4e-3b0f611a58fa\") " pod="ai-tenant-e2e-shared-b-w0-9ed110/auth-test-model-9ed110-kserve-76968d48b9-wlwcm"
Nov 03 09:26:23.895072 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:23.895025    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dc0565c-837f-432a-bf4e-3b0f611a58fa-home\") pod \"auth-test-model-9ed110-kserve-76968d48b9-wlwcm\" (UID: \"9dc0565c-837f-432a-bf4e-3b0f611a58fa\") " pod="ai-tenant-e2e-shared-b-w0-9ed110/auth-test-model-9ed110-kserve-76968d48b9-wlwcm"
Nov 03 09:26:23.896873 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:23.896851    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dc0565c-837f-432a-bf4e-3b0f611a58fa-dshm\") pod \"auth-test-model-9ed110-kserve-76968d48b9-wlwcm\" (UID: \"9dc0565c-837f-432a-bf4e-3b0f611a58fa\") " pod="ai-tenant-e2e-shared-b-w0-9ed110/auth-test-model-9ed110-kserve-76968d48b9-wlwcm"
Nov 03 09:26:23.897129 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:23.897114    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dc0565c-837f-432a-bf4e-3b0f611a58fa-tls-certs\") pod \"auth-test-model-9ed110-kserve-76968d48b9-wlwcm\" (UID: \"9dc0565c-837f-432a-bf4e-3b0f611a58fa\") " pod="ai-tenant-e2e-shared-b-w0-9ed110/auth-test-model-9ed110-kserve-76968d48b9-wlwcm"
Nov 03 09:26:23.958218 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:23.958191    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppzdl\" (UniqueName: \"kubernetes.io/projected/9dc0565c-837f-432a-bf4e-3b0f611a58fa-kube-api-access-ppzdl\") pod \"auth-test-model-9ed110-kserve-76968d48b9-wlwcm\" (UID: \"9dc0565c-837f-432a-bf4e-3b0f611a58fa\") " pod="ai-tenant-e2e-shared-b-w0-9ed110/auth-test-model-9ed110-kserve-76968d48b9-wlwcm"
Nov 03 09:26:24.025096 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:24.025015    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9ed110/auth-test-model-9ed110-kserve-76968d48b9-wlwcm"
Nov 03 09:26:24.419798 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:26:24.419755    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9dc0565c_837f_432a_bf4e_3b0f611a58fa.slice/crio-db4898dfa588c7c2bf772ad29cf85fe79621f71afe372faea2a6f6b6897779dd WatchSource:0}: Error finding container db4898dfa588c7c2bf772ad29cf85fe79621f71afe372faea2a6f6b6897779dd: Status 404 returned error can't find the container with id db4898dfa588c7c2bf772ad29cf85fe79621f71afe372faea2a6f6b6897779dd
Nov 03 09:26:24.420366 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:24.420339    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9ed110/auth-test-model-9ed110-kserve-76968d48b9-wlwcm"]
Nov 03 09:26:24.614815 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:24.614778    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9ed110/auth-test-model-9ed110-kserve-76968d48b9-wlwcm" event={"ID":"9dc0565c-837f-432a-bf4e-3b0f611a58fa","Type":"ContainerStarted","Data":"c5c8564f4150b8febd3bf9e71346d1f198d036cb7296607df6cae603f6e42bd7"}
Nov 03 09:26:24.614815 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:24.614822    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9ed110/auth-test-model-9ed110-kserve-76968d48b9-wlwcm" event={"ID":"9dc0565c-837f-432a-bf4e-3b0f611a58fa","Type":"ContainerStarted","Data":"db4898dfa588c7c2bf772ad29cf85fe79621f71afe372faea2a6f6b6897779dd"}
Nov 03 09:26:24.735312 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:24.735223    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-9ed110/auth-test-model-9ed110-kserve-76968d48b9-wlwcm" podStartSLOduration=1.7352095859999999 podStartE2EDuration="1.735209586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:26:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:26:24.733294798 +0000 UTC m=+2809.002358259" watchObservedRunningTime="2026-11-03 09:26:24.735209586 +0000 UTC m=+2809.004273021"
Nov 03 09:26:25.025756 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:25.025674    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-9ed110/auth-test-model-9ed110-kserve-76968d48b9-wlwcm"
Nov 03 09:26:34.025917 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:34.025883    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-9ed110/auth-test-model-9ed110-kserve-76968d48b9-wlwcm"
Nov 03 09:26:34.038931 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:34.038906    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-9ed110/auth-test-model-9ed110-kserve-76968d48b9-wlwcm"
Nov 03 09:26:34.685600 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:34.685573    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-9ed110/auth-test-model-9ed110-kserve-76968d48b9-wlwcm"
Nov 03 09:26:46.939116 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:26:46.939030    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:26:50.485464 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:50.485434    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-2fc57e/explicit-model-2fc57e-kserve-6ff449c699-sq9dz"]
Nov 03 09:26:53.053763 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:53.053716    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0764e0e9-r7rgm"]
Nov 03 09:26:53.103888 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:53.103847    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0764e0e9-r7rgm"]
Nov 03 09:26:53.871326 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:26:53.871295    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2afd523-012e-4305-9f86-ea7ba8e7d88a" path="/var/lib/kubelet/pods/d2afd523-012e-4305-9f86-ea7ba8e7d88a/volumes"
Nov 03 09:27:05.506796 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:05.506753    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-2fc57e/explicit-model-2fc57e-kserve-6ff449c699-sq9dz" podUID="60a13de7-5fc4-48d8-a1b6-90ac67bdb782" containerName="main" containerID="cri-o://34479bd04e050d05afc8578f095d2e0d6914e05d82f0c3a41db1c1d3d9372340" gracePeriod=45
Nov 03 09:27:05.511149 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:05.511119    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-explicit-2fc57e/explicit-model-2fc57e-kserve-6ff449c699-sq9dz" podUID="60a13de7-5fc4-48d8-a1b6-90ac67bdb782" containerName="main" probeResult="failure" output="Get \"https://10.133.0.164:8000/ready\": EOF"
Nov 03 09:27:05.778247 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:05.778223    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-2fc57e/explicit-model-2fc57e-kserve-6ff449c699-sq9dz"
Nov 03 09:27:05.885065 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:05.885036    4473 generic.go:356] "Generic (PLEG): container finished" podID="60a13de7-5fc4-48d8-a1b6-90ac67bdb782" containerID="34479bd04e050d05afc8578f095d2e0d6914e05d82f0c3a41db1c1d3d9372340" exitCode=0
Nov 03 09:27:05.885223 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:05.885074    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-2fc57e/explicit-model-2fc57e-kserve-6ff449c699-sq9dz"
Nov 03 09:27:05.885223 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:05.885117    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-2fc57e/explicit-model-2fc57e-kserve-6ff449c699-sq9dz" event={"ID":"60a13de7-5fc4-48d8-a1b6-90ac67bdb782","Type":"ContainerDied","Data":"34479bd04e050d05afc8578f095d2e0d6914e05d82f0c3a41db1c1d3d9372340"}
Nov 03 09:27:05.885223 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:05.885153    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-2fc57e/explicit-model-2fc57e-kserve-6ff449c699-sq9dz" event={"ID":"60a13de7-5fc4-48d8-a1b6-90ac67bdb782","Type":"ContainerDied","Data":"bdcaa5da13511741899372c4999219eac935d4aae35788e6ebb84467e9447453"}
Nov 03 09:27:05.885223 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:05.885164    4473 scope.go:122] "RemoveContainer" containerID="34479bd04e050d05afc8578f095d2e0d6914e05d82f0c3a41db1c1d3d9372340"
Nov 03 09:27:05.894277 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:05.894259    4473 scope.go:122] "RemoveContainer" containerID="34479bd04e050d05afc8578f095d2e0d6914e05d82f0c3a41db1c1d3d9372340"
Nov 03 09:27:05.894568 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:27:05.894544    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34479bd04e050d05afc8578f095d2e0d6914e05d82f0c3a41db1c1d3d9372340\": container with ID starting with 34479bd04e050d05afc8578f095d2e0d6914e05d82f0c3a41db1c1d3d9372340 not found: ID does not exist" containerID="34479bd04e050d05afc8578f095d2e0d6914e05d82f0c3a41db1c1d3d9372340"
Nov 03 09:27:05.894652 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:05.894572    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34479bd04e050d05afc8578f095d2e0d6914e05d82f0c3a41db1c1d3d9372340"} err="failed to get container status \"34479bd04e050d05afc8578f095d2e0d6914e05d82f0c3a41db1c1d3d9372340\": rpc error: code = NotFound desc = could not find container \"34479bd04e050d05afc8578f095d2e0d6914e05d82f0c3a41db1c1d3d9372340\": container with ID starting with 34479bd04e050d05afc8578f095d2e0d6914e05d82f0c3a41db1c1d3d9372340 not found: ID does not exist"
Nov 03 09:27:05.918844 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:05.918805    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-kube-api-access-pbhq8\" (UniqueName: \"kubernetes.io/projected/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-kube-api-access-pbhq8\") pod \"60a13de7-5fc4-48d8-a1b6-90ac67bdb782\" (UID: \"60a13de7-5fc4-48d8-a1b6-90ac67bdb782\") "
Nov 03 09:27:05.919022 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:05.918899    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-tls-certs\" (UniqueName: \"kubernetes.io/secret/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-tls-certs\") pod \"60a13de7-5fc4-48d8-a1b6-90ac67bdb782\" (UID: \"60a13de7-5fc4-48d8-a1b6-90ac67bdb782\") "
Nov 03 09:27:05.919022 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:05.918986    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-dshm\" (UniqueName: \"kubernetes.io/empty-dir/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-dshm\") pod \"60a13de7-5fc4-48d8-a1b6-90ac67bdb782\" (UID: \"60a13de7-5fc4-48d8-a1b6-90ac67bdb782\") "
Nov 03 09:27:05.919105 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:05.919041    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-tmp-dir\") pod \"60a13de7-5fc4-48d8-a1b6-90ac67bdb782\" (UID: \"60a13de7-5fc4-48d8-a1b6-90ac67bdb782\") "
Nov 03 09:27:05.919199 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:05.919183    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-model-cache\") pod \"60a13de7-5fc4-48d8-a1b6-90ac67bdb782\" (UID: \"60a13de7-5fc4-48d8-a1b6-90ac67bdb782\") "
Nov 03 09:27:05.919247 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:05.919233    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-home\" (UniqueName: \"kubernetes.io/empty-dir/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-home\") pod \"60a13de7-5fc4-48d8-a1b6-90ac67bdb782\" (UID: \"60a13de7-5fc4-48d8-a1b6-90ac67bdb782\") "
Nov 03 09:27:05.920771 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:05.920587    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-model-cache" pod "60a13de7-5fc4-48d8-a1b6-90ac67bdb782" (UID: "60a13de7-5fc4-48d8-a1b6-90ac67bdb782"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:27:05.920771 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:05.920653    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-home" pod "60a13de7-5fc4-48d8-a1b6-90ac67bdb782" (UID: "60a13de7-5fc4-48d8-a1b6-90ac67bdb782"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:27:05.921088 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:05.921049    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-tmp-dir" pod "60a13de7-5fc4-48d8-a1b6-90ac67bdb782" (UID: "60a13de7-5fc4-48d8-a1b6-90ac67bdb782"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:27:05.922657 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:05.922416    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-tls-certs" pod "60a13de7-5fc4-48d8-a1b6-90ac67bdb782" (UID: "60a13de7-5fc4-48d8-a1b6-90ac67bdb782"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:27:05.924420 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:05.924396    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-dshm" pod "60a13de7-5fc4-48d8-a1b6-90ac67bdb782" (UID: "60a13de7-5fc4-48d8-a1b6-90ac67bdb782"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:27:05.924680 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:05.924654    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-kube-api-access-pbhq8" pod "60a13de7-5fc4-48d8-a1b6-90ac67bdb782" (UID: "60a13de7-5fc4-48d8-a1b6-90ac67bdb782"). InnerVolumeSpecName "kube-api-access-pbhq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:27:06.020763 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:06.020674    4473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-model-cache\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:27:06.020763 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:06.020717    4473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-home\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:27:06.020763 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:06.020736    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbhq8\" (UniqueName: \"kubernetes.io/projected/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-kube-api-access-pbhq8\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:27:06.020763 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:06.020751    4473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-tls-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:27:06.020763 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:06.020760    4473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-dshm\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:27:06.020763 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:06.020768    4473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60a13de7-5fc4-48d8-a1b6-90ac67bdb782-tmp-dir\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:27:06.341838 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:06.341724    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-2fc57e/explicit-model-2fc57e-kserve-6ff449c699-sq9dz"]
Nov 03 09:27:06.345849 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:06.345816    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-2fc57e/explicit-model-2fc57e-kserve-6ff449c699-sq9dz"]
Nov 03 09:27:07.870539 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:07.870503    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60a13de7-5fc4-48d8-a1b6-90ac67bdb782" path="/var/lib/kubelet/pods/60a13de7-5fc4-48d8-a1b6-90ac67bdb782/volumes"
Nov 03 09:27:16.937400 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:27:16.937366    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:27:19.921777 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:19.921740    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:27:26.966744 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:26.966707    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-2fc57e-b79446cf5-sfm7v"]
Nov 03 09:27:31.967869 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:31.967829    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2fc57e-b79446cf5-sfm7v" podUID="60535aee-30e5-4b8f-a7e3-733e3b933a67" containerName="payload-pre-processing" containerID="cri-o://5c3775e2a7a7f18623a3804833ce3eb7c43c1fa2c3e1ea32cb9598b5e2e365fe" gracePeriod=25
Nov 03 09:27:32.229753 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:32.229701    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2fc57e-b79446cf5-sfm7v"
Nov 03 09:27:32.263347 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:32.263320    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60535aee-30e5-4b8f-a7e3-733e3b933a67-kube-api-access-896ql\" (UniqueName: \"kubernetes.io/projected/60535aee-30e5-4b8f-a7e3-733e3b933a67-kube-api-access-896ql\") pod \"60535aee-30e5-4b8f-a7e3-733e3b933a67\" (UID: \"60535aee-30e5-4b8f-a7e3-733e3b933a67\") "
Nov 03 09:27:32.263507 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:32.263486    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60535aee-30e5-4b8f-a7e3-733e3b933a67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60535aee-30e5-4b8f-a7e3-733e3b933a67-plugins-config-volume\") pod \"60535aee-30e5-4b8f-a7e3-733e3b933a67\" (UID: \"60535aee-30e5-4b8f-a7e3-733e3b933a67\") "
Nov 03 09:27:32.263813 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:32.263790    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60535aee-30e5-4b8f-a7e3-733e3b933a67-plugins-config-volume" pod "60535aee-30e5-4b8f-a7e3-733e3b933a67" (UID: "60535aee-30e5-4b8f-a7e3-733e3b933a67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:27:32.263976 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:32.263938    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60535aee-30e5-4b8f-a7e3-733e3b933a67-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:27:32.265658 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:32.265640    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60535aee-30e5-4b8f-a7e3-733e3b933a67-kube-api-access-896ql" pod "60535aee-30e5-4b8f-a7e3-733e3b933a67" (UID: "60535aee-30e5-4b8f-a7e3-733e3b933a67"). InnerVolumeSpecName "kube-api-access-896ql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:27:32.365314 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:32.365287    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-896ql\" (UniqueName: \"kubernetes.io/projected/60535aee-30e5-4b8f-a7e3-733e3b933a67-kube-api-access-896ql\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:27:33.047787 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:33.047752    4473 generic.go:356] "Generic (PLEG): container finished" podID="60535aee-30e5-4b8f-a7e3-733e3b933a67" containerID="5c3775e2a7a7f18623a3804833ce3eb7c43c1fa2c3e1ea32cb9598b5e2e365fe" exitCode=0
Nov 03 09:27:33.048238 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:33.047805    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2fc57e-b79446cf5-sfm7v"
Nov 03 09:27:33.048238 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:33.047845    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2fc57e-b79446cf5-sfm7v" event={"ID":"60535aee-30e5-4b8f-a7e3-733e3b933a67","Type":"ContainerDied","Data":"5c3775e2a7a7f18623a3804833ce3eb7c43c1fa2c3e1ea32cb9598b5e2e365fe"}
Nov 03 09:27:33.048238 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:33.047924    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2fc57e-b79446cf5-sfm7v" event={"ID":"60535aee-30e5-4b8f-a7e3-733e3b933a67","Type":"ContainerDied","Data":"d16ab5554a2d37a40ec9cbd96d57bd8fe73b1ed9c9c206fdac0c0b0c6b4760c1"}
Nov 03 09:27:33.048238 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:33.047961    4473 scope.go:122] "RemoveContainer" containerID="5c3775e2a7a7f18623a3804833ce3eb7c43c1fa2c3e1ea32cb9598b5e2e365fe"
Nov 03 09:27:33.057659 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:33.057644    4473 scope.go:122] "RemoveContainer" containerID="5c3775e2a7a7f18623a3804833ce3eb7c43c1fa2c3e1ea32cb9598b5e2e365fe"
Nov 03 09:27:33.057922 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:27:33.057903    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c3775e2a7a7f18623a3804833ce3eb7c43c1fa2c3e1ea32cb9598b5e2e365fe\": container with ID starting with 5c3775e2a7a7f18623a3804833ce3eb7c43c1fa2c3e1ea32cb9598b5e2e365fe not found: ID does not exist" containerID="5c3775e2a7a7f18623a3804833ce3eb7c43c1fa2c3e1ea32cb9598b5e2e365fe"
Nov 03 09:27:33.058024 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:33.057930    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c3775e2a7a7f18623a3804833ce3eb7c43c1fa2c3e1ea32cb9598b5e2e365fe"} err="failed to get container status \"5c3775e2a7a7f18623a3804833ce3eb7c43c1fa2c3e1ea32cb9598b5e2e365fe\": rpc error: code = NotFound desc = could not find container \"5c3775e2a7a7f18623a3804833ce3eb7c43c1fa2c3e1ea32cb9598b5e2e365fe\": container with ID starting with 5c3775e2a7a7f18623a3804833ce3eb7c43c1fa2c3e1ea32cb9598b5e2e365fe not found: ID does not exist"
Nov 03 09:27:33.216217 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:33.216183    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-2fc57e-b79446cf5-sfm7v"]
Nov 03 09:27:33.223237 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:33.223204    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-2fc57e-b79446cf5-sfm7v"]
Nov 03 09:27:33.870959 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:33.870921    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60535aee-30e5-4b8f-a7e3-733e3b933a67" path="/var/lib/kubelet/pods/60535aee-30e5-4b8f-a7e3-733e3b933a67/volumes"
Nov 03 09:27:43.967583 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:43.967549    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"]
Nov 03 09:27:43.968067 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:43.967766    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq" podUID="a0cc45ff-b2a8-408b-9434-812476948826" containerName="istio-proxy" containerID="cri-o://3d30a47e1d3a192db76a92a4b3f530ddfa0f04045eeca9995de5e75ad85c0945" gracePeriod=30
Nov 03 09:27:46.937717 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:27:46.937678    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:27:49.148124 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:49.148086    4473 generic.go:356] "Generic (PLEG): container finished" podID="a0cc45ff-b2a8-408b-9434-812476948826" containerID="3d30a47e1d3a192db76a92a4b3f530ddfa0f04045eeca9995de5e75ad85c0945" exitCode=0
Nov 03 09:27:49.148519 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:49.148162    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq" event={"ID":"a0cc45ff-b2a8-408b-9434-812476948826","Type":"ContainerDied","Data":"3d30a47e1d3a192db76a92a4b3f530ddfa0f04045eeca9995de5e75ad85c0945"}
Nov 03 09:27:49.148519 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:49.148242    4473 scope.go:122] "RemoveContainer" containerID="b2f3d4be138ef4d26a1147dd59695a0aa4350e0b7757865108d05a43d6b351f6"
Nov 03 09:27:49.224008 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:49.223986    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"
Nov 03 09:27:49.365705 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:49.365681    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0cc45ff-b2a8-408b-9434-812476948826-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0cc45ff-b2a8-408b-9434-812476948826-istio-data\") pod \"a0cc45ff-b2a8-408b-9434-812476948826\" (UID: \"a0cc45ff-b2a8-408b-9434-812476948826\") "
Nov 03 09:27:49.365865 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:49.365724    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0cc45ff-b2a8-408b-9434-812476948826-istio-token\" (UniqueName: \"kubernetes.io/projected/a0cc45ff-b2a8-408b-9434-812476948826-istio-token\") pod \"a0cc45ff-b2a8-408b-9434-812476948826\" (UID: \"a0cc45ff-b2a8-408b-9434-812476948826\") "
Nov 03 09:27:49.365865 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:49.365750    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0cc45ff-b2a8-408b-9434-812476948826-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0cc45ff-b2a8-408b-9434-812476948826-istiod-ca-cert\") pod \"a0cc45ff-b2a8-408b-9434-812476948826\" (UID: \"a0cc45ff-b2a8-408b-9434-812476948826\") "
Nov 03 09:27:49.365865 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:49.365787    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0cc45ff-b2a8-408b-9434-812476948826-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0cc45ff-b2a8-408b-9434-812476948826-istio-podinfo\") pod \"a0cc45ff-b2a8-408b-9434-812476948826\" (UID: \"a0cc45ff-b2a8-408b-9434-812476948826\") "
Nov 03 09:27:49.365865 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:49.365861    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0cc45ff-b2a8-408b-9434-812476948826-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0cc45ff-b2a8-408b-9434-812476948826-credential-socket\") pod \"a0cc45ff-b2a8-408b-9434-812476948826\" (UID: \"a0cc45ff-b2a8-408b-9434-812476948826\") "
Nov 03 09:27:49.366132 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:49.365922    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0cc45ff-b2a8-408b-9434-812476948826-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0cc45ff-b2a8-408b-9434-812476948826-istio-envoy\") pod \"a0cc45ff-b2a8-408b-9434-812476948826\" (UID: \"a0cc45ff-b2a8-408b-9434-812476948826\") "
Nov 03 09:27:49.366132 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:49.366016    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0cc45ff-b2a8-408b-9434-812476948826-istio-data" pod "a0cc45ff-b2a8-408b-9434-812476948826" (UID: "a0cc45ff-b2a8-408b-9434-812476948826"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:27:49.366132 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:49.366023    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0cc45ff-b2a8-408b-9434-812476948826-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0cc45ff-b2a8-408b-9434-812476948826-workload-certs\") pod \"a0cc45ff-b2a8-408b-9434-812476948826\" (UID: \"a0cc45ff-b2a8-408b-9434-812476948826\") "
Nov 03 09:27:49.366132 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:49.366123    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0cc45ff-b2a8-408b-9434-812476948826-kube-api-access-r5q2f\" (UniqueName: \"kubernetes.io/projected/a0cc45ff-b2a8-408b-9434-812476948826-kube-api-access-r5q2f\") pod \"a0cc45ff-b2a8-408b-9434-812476948826\" (UID: \"a0cc45ff-b2a8-408b-9434-812476948826\") "
Nov 03 09:27:49.366355 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:49.366239    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0cc45ff-b2a8-408b-9434-812476948826-workload-certs" pod "a0cc45ff-b2a8-408b-9434-812476948826" (UID: "a0cc45ff-b2a8-408b-9434-812476948826"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:27:49.366355 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:49.366241    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0cc45ff-b2a8-408b-9434-812476948826-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0cc45ff-b2a8-408b-9434-812476948826-workload-socket\") pod \"a0cc45ff-b2a8-408b-9434-812476948826\" (UID: \"a0cc45ff-b2a8-408b-9434-812476948826\") "
Nov 03 09:27:49.366355 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:49.366256    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0cc45ff-b2a8-408b-9434-812476948826-credential-socket" pod "a0cc45ff-b2a8-408b-9434-812476948826" (UID: "a0cc45ff-b2a8-408b-9434-812476948826"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:27:49.366355 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:49.366305    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0cc45ff-b2a8-408b-9434-812476948826-istiod-ca-cert" pod "a0cc45ff-b2a8-408b-9434-812476948826" (UID: "a0cc45ff-b2a8-408b-9434-812476948826"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:27:49.366585 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:49.366430    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0cc45ff-b2a8-408b-9434-812476948826-workload-socket" pod "a0cc45ff-b2a8-408b-9434-812476948826" (UID: "a0cc45ff-b2a8-408b-9434-812476948826"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:27:49.366994 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:49.366971    4473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0cc45ff-b2a8-408b-9434-812476948826-credential-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:27:49.367088 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:49.367000    4473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0cc45ff-b2a8-408b-9434-812476948826-workload-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:27:49.367088 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:49.367017    4473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0cc45ff-b2a8-408b-9434-812476948826-workload-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:27:49.367088 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:49.367031    4473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0cc45ff-b2a8-408b-9434-812476948826-istio-data\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:27:49.367088 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:49.367046    4473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0cc45ff-b2a8-408b-9434-812476948826-istiod-ca-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:27:49.368184 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:49.368156    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0cc45ff-b2a8-408b-9434-812476948826-istio-token" pod "a0cc45ff-b2a8-408b-9434-812476948826" (UID: "a0cc45ff-b2a8-408b-9434-812476948826"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:27:49.368278 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:49.368224    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0cc45ff-b2a8-408b-9434-812476948826-istio-podinfo" pod "a0cc45ff-b2a8-408b-9434-812476948826" (UID: "a0cc45ff-b2a8-408b-9434-812476948826"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 09:27:49.368345 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:49.368314    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0cc45ff-b2a8-408b-9434-812476948826-istio-envoy" pod "a0cc45ff-b2a8-408b-9434-812476948826" (UID: "a0cc45ff-b2a8-408b-9434-812476948826"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:27:49.368496 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:49.368478    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0cc45ff-b2a8-408b-9434-812476948826-kube-api-access-r5q2f" pod "a0cc45ff-b2a8-408b-9434-812476948826" (UID: "a0cc45ff-b2a8-408b-9434-812476948826"). InnerVolumeSpecName "kube-api-access-r5q2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:27:49.468615 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:49.468575    4473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0cc45ff-b2a8-408b-9434-812476948826-istio-token\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:27:49.468615 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:49.468616    4473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0cc45ff-b2a8-408b-9434-812476948826-istio-podinfo\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:27:49.468895 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:49.468632    4473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0cc45ff-b2a8-408b-9434-812476948826-istio-envoy\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:27:49.468895 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:49.468646    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5q2f\" (UniqueName: \"kubernetes.io/projected/a0cc45ff-b2a8-408b-9434-812476948826-kube-api-access-r5q2f\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:27:50.157304 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:50.157222    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"
Nov 03 09:27:50.157740 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:50.157218    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq" event={"ID":"a0cc45ff-b2a8-408b-9434-812476948826","Type":"ContainerDied","Data":"c07c54532b30baf4ced3c70861620f3dd25adf262552be0a8f71af7b3c736abc"}
Nov 03 09:27:50.157740 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:50.157367    4473 scope.go:122] "RemoveContainer" containerID="3d30a47e1d3a192db76a92a4b3f530ddfa0f04045eeca9995de5e75ad85c0945"
Nov 03 09:27:50.326303 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:50.326270    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"]
Nov 03 09:27:50.376654 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:50.376615    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-pfczq"]
Nov 03 09:27:51.869409 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:27:51.869376    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0cc45ff-b2a8-408b-9434-812476948826" path="/var/lib/kubelet/pods/a0cc45ff-b2a8-408b-9434-812476948826/volumes"
Nov 03 09:28:15.364700 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:15.364651    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:28:16.257534 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:16.254565    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2fc57e-v7cb8"]
Nov 03 09:28:16.257534 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:16.255666    4473 cpu_manager.go:414] "containerMap: removing container" podUID="60a13de7-5fc4-48d8-a1b6-90ac67bdb782" containerName="main"
Nov 03 09:28:16.257534 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:16.255687    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60a13de7-5fc4-48d8-a1b6-90ac67bdb782" containerName="main"
Nov 03 09:28:16.257534 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:16.255756    4473 cpu_manager.go:414] "containerMap: removing container" podUID="a0cc45ff-b2a8-408b-9434-812476948826" containerName="istio-proxy"
Nov 03 09:28:16.257534 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:16.255764    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0cc45ff-b2a8-408b-9434-812476948826" containerName="istio-proxy"
Nov 03 09:28:16.257534 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:16.255784    4473 cpu_manager.go:414] "containerMap: removing container" podUID="a0cc45ff-b2a8-408b-9434-812476948826" containerName="istio-proxy"
Nov 03 09:28:16.257534 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:16.255792    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0cc45ff-b2a8-408b-9434-812476948826" containerName="istio-proxy"
Nov 03 09:28:16.257534 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:16.255834    4473 cpu_manager.go:414] "containerMap: removing container" podUID="60535aee-30e5-4b8f-a7e3-733e3b933a67" containerName="payload-pre-processing"
Nov 03 09:28:16.257534 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:16.255841    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60535aee-30e5-4b8f-a7e3-733e3b933a67" containerName="payload-pre-processing"
Nov 03 09:28:16.257534 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:16.256117    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="60a13de7-5fc4-48d8-a1b6-90ac67bdb782" containerName="main"
Nov 03 09:28:16.257534 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:16.256139    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0cc45ff-b2a8-408b-9434-812476948826" containerName="istio-proxy"
Nov 03 09:28:16.257534 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:16.256157    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0cc45ff-b2a8-408b-9434-812476948826" containerName="istio-proxy"
Nov 03 09:28:16.257534 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:16.256175    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="60535aee-30e5-4b8f-a7e3-733e3b933a67" containerName="payload-pre-processing"
Nov 03 09:28:16.260379 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:16.260355    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2fc57e-v7cb8"
Nov 03 09:28:16.308795 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:16.308773    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2fc57e-v7cb8"]
Nov 03 09:28:16.355476 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:16.355438    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:28:16.355476 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:16.355473    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:28:16.355708 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:16.355500    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:28:16.363919 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:16.363897    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:28:16.393712 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:16.393686    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkszz\" (UniqueName: \"kubernetes.io/projected/c2ba685c-7259-47fe-80f4-8cab8e797b51-kube-api-access-vkszz\") pod \"maas-api-revoke-keys-e2e-explicit-2fc57e-v7cb8\" (UID: \"c2ba685c-7259-47fe-80f4-8cab8e797b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2fc57e-v7cb8"
Nov 03 09:28:16.394090 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:16.393714    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2ba685c-7259-47fe-80f4-8cab8e797b51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-2fc57e-v7cb8\" (UID: \"c2ba685c-7259-47fe-80f4-8cab8e797b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2fc57e-v7cb8"
Nov 03 09:28:16.394090 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:16.393836    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2ba685c-7259-47fe-80f4-8cab8e797b51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-2fc57e-v7cb8\" (UID: \"c2ba685c-7259-47fe-80f4-8cab8e797b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2fc57e-v7cb8"
Nov 03 09:28:16.495489 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:16.495445    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkszz\" (UniqueName: \"kubernetes.io/projected/c2ba685c-7259-47fe-80f4-8cab8e797b51-kube-api-access-vkszz\") pod \"maas-api-revoke-keys-e2e-explicit-2fc57e-v7cb8\" (UID: \"c2ba685c-7259-47fe-80f4-8cab8e797b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2fc57e-v7cb8"
Nov 03 09:28:16.495489 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:16.495492    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2ba685c-7259-47fe-80f4-8cab8e797b51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-2fc57e-v7cb8\" (UID: \"c2ba685c-7259-47fe-80f4-8cab8e797b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2fc57e-v7cb8"
Nov 03 09:28:16.495733 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:16.495540    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2ba685c-7259-47fe-80f4-8cab8e797b51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-2fc57e-v7cb8\" (UID: \"c2ba685c-7259-47fe-80f4-8cab8e797b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2fc57e-v7cb8"
Nov 03 09:28:16.496198 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:16.496180    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2ba685c-7259-47fe-80f4-8cab8e797b51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-2fc57e-v7cb8\" (UID: \"c2ba685c-7259-47fe-80f4-8cab8e797b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2fc57e-v7cb8"
Nov 03 09:28:16.498003 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:16.497936    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2ba685c-7259-47fe-80f4-8cab8e797b51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-2fc57e-v7cb8\" (UID: \"c2ba685c-7259-47fe-80f4-8cab8e797b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2fc57e-v7cb8"
Nov 03 09:28:16.557218 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:16.557155    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkszz\" (UniqueName: \"kubernetes.io/projected/c2ba685c-7259-47fe-80f4-8cab8e797b51-kube-api-access-vkszz\") pod \"maas-api-revoke-keys-e2e-explicit-2fc57e-v7cb8\" (UID: \"c2ba685c-7259-47fe-80f4-8cab8e797b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2fc57e-v7cb8"
Nov 03 09:28:16.574116 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:16.574089    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2fc57e-v7cb8"
Nov 03 09:28:16.937421 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:28:16.937351    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:28:16.964087 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:16.964060    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2fc57e-v7cb8"]
Nov 03 09:28:16.965558 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:28:16.965535    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2ba685c_7259_47fe_80f4_8cab8e797b51.slice/crio-0ab4c1a1a62274757ba29767e99e763e5f5107f2283b600071e1012d898f3f76 WatchSource:0}: Error finding container 0ab4c1a1a62274757ba29767e99e763e5f5107f2283b600071e1012d898f3f76: Status 404 returned error can't find the container with id 0ab4c1a1a62274757ba29767e99e763e5f5107f2283b600071e1012d898f3f76
Nov 03 09:28:17.317685 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:17.317649    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2fc57e-v7cb8" event={"ID":"c2ba685c-7259-47fe-80f4-8cab8e797b51","Type":"ContainerStarted","Data":"c6a843ce30f40bcd9fbcb3c4b2e41a3ddcbe68eb43257dc64bcd6eeadfd4f94a"}
Nov 03 09:28:17.317832 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:17.317719    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2fc57e-v7cb8" event={"ID":"c2ba685c-7259-47fe-80f4-8cab8e797b51","Type":"ContainerStarted","Data":"0ab4c1a1a62274757ba29767e99e763e5f5107f2283b600071e1012d898f3f76"}
Nov 03 09:28:17.440912 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:17.440864    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2fc57e-v7cb8" podStartSLOduration=1.440849982 podStartE2EDuration="1.440849982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:28:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:28:17.438428571 +0000 UTC m=+2921.707492002" watchObservedRunningTime="2026-11-03 09:28:17.440849982 +0000 UTC m=+2921.709913416"
Nov 03 09:28:18.324773 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:18.324736    4473 generic.go:356] "Generic (PLEG): container finished" podID="c2ba685c-7259-47fe-80f4-8cab8e797b51" containerID="c6a843ce30f40bcd9fbcb3c4b2e41a3ddcbe68eb43257dc64bcd6eeadfd4f94a" exitCode=0
Nov 03 09:28:18.324984 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:18.324827    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2fc57e-v7cb8" event={"ID":"c2ba685c-7259-47fe-80f4-8cab8e797b51","Type":"ContainerDied","Data":"c6a843ce30f40bcd9fbcb3c4b2e41a3ddcbe68eb43257dc64bcd6eeadfd4f94a"}
Nov 03 09:28:19.479222 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:19.479201    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2fc57e-v7cb8"
Nov 03 09:28:19.527233 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:19.527205    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2ba685c-7259-47fe-80f4-8cab8e797b51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2ba685c-7259-47fe-80f4-8cab8e797b51-maas-api-xks-ca\") pod \"c2ba685c-7259-47fe-80f4-8cab8e797b51\" (UID: \"c2ba685c-7259-47fe-80f4-8cab8e797b51\") "
Nov 03 09:28:19.527423 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:19.527306    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2ba685c-7259-47fe-80f4-8cab8e797b51-kube-api-access-vkszz\" (UniqueName: \"kubernetes.io/projected/c2ba685c-7259-47fe-80f4-8cab8e797b51-kube-api-access-vkszz\") pod \"c2ba685c-7259-47fe-80f4-8cab8e797b51\" (UID: \"c2ba685c-7259-47fe-80f4-8cab8e797b51\") "
Nov 03 09:28:19.527497 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:19.527419    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2ba685c-7259-47fe-80f4-8cab8e797b51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2ba685c-7259-47fe-80f4-8cab8e797b51-maas-api-openshift-ca\") pod \"c2ba685c-7259-47fe-80f4-8cab8e797b51\" (UID: \"c2ba685c-7259-47fe-80f4-8cab8e797b51\") "
Nov 03 09:28:19.527742 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:19.527720    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2ba685c-7259-47fe-80f4-8cab8e797b51-maas-api-openshift-ca" pod "c2ba685c-7259-47fe-80f4-8cab8e797b51" (UID: "c2ba685c-7259-47fe-80f4-8cab8e797b51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:28:19.527862 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:19.527834    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2ba685c-7259-47fe-80f4-8cab8e797b51-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:28:19.529444 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:19.529426    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2ba685c-7259-47fe-80f4-8cab8e797b51-maas-api-xks-ca" pod "c2ba685c-7259-47fe-80f4-8cab8e797b51" (UID: "c2ba685c-7259-47fe-80f4-8cab8e797b51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:28:19.529517 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:19.529477    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ba685c-7259-47fe-80f4-8cab8e797b51-kube-api-access-vkszz" pod "c2ba685c-7259-47fe-80f4-8cab8e797b51" (UID: "c2ba685c-7259-47fe-80f4-8cab8e797b51"). InnerVolumeSpecName "kube-api-access-vkszz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:28:19.628809 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:19.628773    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkszz\" (UniqueName: \"kubernetes.io/projected/c2ba685c-7259-47fe-80f4-8cab8e797b51-kube-api-access-vkszz\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:28:19.628809 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:19.628799    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2ba685c-7259-47fe-80f4-8cab8e797b51-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:28:20.341847 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:20.341809    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2fc57e-v7cb8" event={"ID":"c2ba685c-7259-47fe-80f4-8cab8e797b51","Type":"ContainerDied","Data":"0ab4c1a1a62274757ba29767e99e763e5f5107f2283b600071e1012d898f3f76"}
Nov 03 09:28:20.341847 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:20.341849    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2fc57e-v7cb8"
Nov 03 09:28:20.342119 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:20.341860    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ab4c1a1a62274757ba29767e99e763e5f5107f2283b600071e1012d898f3f76"
Nov 03 09:28:31.732180 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:31.732148    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2fc57e-v7cb8"]
Nov 03 09:28:31.782229 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:31.782193    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2fc57e-v7cb8"]
Nov 03 09:28:31.870154 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:31.870122    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2ba685c-7259-47fe-80f4-8cab8e797b51" path="/var/lib/kubelet/pods/c2ba685c-7259-47fe-80f4-8cab8e797b51/volumes"
Nov 03 09:28:38.121659 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:38.121618    4473 scope.go:122] "RemoveContainer" containerID="0b85a48546eb6bfb2a93a5a73017556bda2b139936a6ed30a4332b81dba03bd3"
Nov 03 09:28:42.233696 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:42.233666    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b46dc8f6-t8tvg"]
Nov 03 09:28:42.234184 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:42.234169    4473 cpu_manager.go:414] "containerMap: removing container" podUID="c2ba685c-7259-47fe-80f4-8cab8e797b51" containerName="revoke-keys"
Nov 03 09:28:42.234229 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:42.234185    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2ba685c-7259-47fe-80f4-8cab8e797b51" containerName="revoke-keys"
Nov 03 09:28:42.234294 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:42.234285    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2ba685c-7259-47fe-80f4-8cab8e797b51" containerName="revoke-keys"
Nov 03 09:28:42.238730 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:42.238714    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b46dc8f6-t8tvg"
Nov 03 09:28:42.294213 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:42.294188    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:28:42.294213 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:42.294198    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:28:42.294213 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:42.294224    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:28:42.296576 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:42.296548    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b46dc8f6-t8tvg"]
Nov 03 09:28:42.343577 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:42.343553    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:28:42.404236 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:42.404203    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87ddf68f-44eb-473c-aa9d-785716cde3a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b46dc8f6-t8tvg\" (UID: \"87ddf68f-44eb-473c-aa9d-785716cde3a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b46dc8f6-t8tvg"
Nov 03 09:28:42.404380 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:42.404260    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbwhd\" (UniqueName: \"kubernetes.io/projected/87ddf68f-44eb-473c-aa9d-785716cde3a6-kube-api-access-lbwhd\") pod \"maas-api-revoke-sub-b46dc8f6-t8tvg\" (UID: \"87ddf68f-44eb-473c-aa9d-785716cde3a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b46dc8f6-t8tvg"
Nov 03 09:28:42.404380 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:42.404346    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87ddf68f-44eb-473c-aa9d-785716cde3a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b46dc8f6-t8tvg\" (UID: \"87ddf68f-44eb-473c-aa9d-785716cde3a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b46dc8f6-t8tvg"
Nov 03 09:28:42.505907 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:42.505835    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87ddf68f-44eb-473c-aa9d-785716cde3a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b46dc8f6-t8tvg\" (UID: \"87ddf68f-44eb-473c-aa9d-785716cde3a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b46dc8f6-t8tvg"
Nov 03 09:28:42.505907 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:42.505886    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbwhd\" (UniqueName: \"kubernetes.io/projected/87ddf68f-44eb-473c-aa9d-785716cde3a6-kube-api-access-lbwhd\") pod \"maas-api-revoke-sub-b46dc8f6-t8tvg\" (UID: \"87ddf68f-44eb-473c-aa9d-785716cde3a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b46dc8f6-t8tvg"
Nov 03 09:28:42.506094 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:42.505908    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87ddf68f-44eb-473c-aa9d-785716cde3a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b46dc8f6-t8tvg\" (UID: \"87ddf68f-44eb-473c-aa9d-785716cde3a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b46dc8f6-t8tvg"
Nov 03 09:28:42.506474 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:42.506455    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87ddf68f-44eb-473c-aa9d-785716cde3a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b46dc8f6-t8tvg\" (UID: \"87ddf68f-44eb-473c-aa9d-785716cde3a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b46dc8f6-t8tvg"
Nov 03 09:28:42.508389 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:42.508361    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87ddf68f-44eb-473c-aa9d-785716cde3a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b46dc8f6-t8tvg\" (UID: \"87ddf68f-44eb-473c-aa9d-785716cde3a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b46dc8f6-t8tvg"
Nov 03 09:28:42.565453 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:42.565420    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbwhd\" (UniqueName: \"kubernetes.io/projected/87ddf68f-44eb-473c-aa9d-785716cde3a6-kube-api-access-lbwhd\") pod \"maas-api-revoke-sub-b46dc8f6-t8tvg\" (UID: \"87ddf68f-44eb-473c-aa9d-785716cde3a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b46dc8f6-t8tvg"
Nov 03 09:28:42.853357 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:42.853273    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b46dc8f6-t8tvg"
Nov 03 09:28:43.235542 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:43.235513    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b46dc8f6-t8tvg"]
Nov 03 09:28:43.236544 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:28:43.236514    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87ddf68f_44eb_473c_aa9d_785716cde3a6.slice/crio-0a47e4c805421b697924b2aef2aa204088af7c389816bb6f496ed8c6e0499bd0 WatchSource:0}: Error finding container 0a47e4c805421b697924b2aef2aa204088af7c389816bb6f496ed8c6e0499bd0: Status 404 returned error can't find the container with id 0a47e4c805421b697924b2aef2aa204088af7c389816bb6f496ed8c6e0499bd0
Nov 03 09:28:43.481760 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:43.481731    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b46dc8f6-t8tvg" event={"ID":"87ddf68f-44eb-473c-aa9d-785716cde3a6","Type":"ContainerStarted","Data":"0a47e4c805421b697924b2aef2aa204088af7c389816bb6f496ed8c6e0499bd0"}
Nov 03 09:28:44.489362 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:44.489287    4473 generic.go:356] "Generic (PLEG): container finished" podID="87ddf68f-44eb-473c-aa9d-785716cde3a6" containerID="9ecce1799df0b8f5d78f041f66de83b161e180bec6fbbdc376ec85dfaf88dec2" exitCode=0
Nov 03 09:28:44.489748 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:44.489367    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b46dc8f6-t8tvg" event={"ID":"87ddf68f-44eb-473c-aa9d-785716cde3a6","Type":"ContainerDied","Data":"9ecce1799df0b8f5d78f041f66de83b161e180bec6fbbdc376ec85dfaf88dec2"}
Nov 03 09:28:45.645760 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:45.645734    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b46dc8f6-t8tvg"
Nov 03 09:28:45.747528 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:45.747457    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87ddf68f-44eb-473c-aa9d-785716cde3a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87ddf68f-44eb-473c-aa9d-785716cde3a6-maas-api-xks-ca\") pod \"87ddf68f-44eb-473c-aa9d-785716cde3a6\" (UID: \"87ddf68f-44eb-473c-aa9d-785716cde3a6\") "
Nov 03 09:28:45.747528 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:45.747522    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87ddf68f-44eb-473c-aa9d-785716cde3a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87ddf68f-44eb-473c-aa9d-785716cde3a6-maas-api-openshift-ca\") pod \"87ddf68f-44eb-473c-aa9d-785716cde3a6\" (UID: \"87ddf68f-44eb-473c-aa9d-785716cde3a6\") "
Nov 03 09:28:45.747716 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:45.747587    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87ddf68f-44eb-473c-aa9d-785716cde3a6-kube-api-access-lbwhd\" (UniqueName: \"kubernetes.io/projected/87ddf68f-44eb-473c-aa9d-785716cde3a6-kube-api-access-lbwhd\") pod \"87ddf68f-44eb-473c-aa9d-785716cde3a6\" (UID: \"87ddf68f-44eb-473c-aa9d-785716cde3a6\") "
Nov 03 09:28:45.747816 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:45.747797    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87ddf68f-44eb-473c-aa9d-785716cde3a6-maas-api-openshift-ca" pod "87ddf68f-44eb-473c-aa9d-785716cde3a6" (UID: "87ddf68f-44eb-473c-aa9d-785716cde3a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:28:45.748074 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:45.748053    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87ddf68f-44eb-473c-aa9d-785716cde3a6-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:28:45.749802 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:45.749772    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87ddf68f-44eb-473c-aa9d-785716cde3a6-maas-api-xks-ca" pod "87ddf68f-44eb-473c-aa9d-785716cde3a6" (UID: "87ddf68f-44eb-473c-aa9d-785716cde3a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:28:45.750174 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:45.750151    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ddf68f-44eb-473c-aa9d-785716cde3a6-kube-api-access-lbwhd" pod "87ddf68f-44eb-473c-aa9d-785716cde3a6" (UID: "87ddf68f-44eb-473c-aa9d-785716cde3a6"). InnerVolumeSpecName "kube-api-access-lbwhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:28:45.849179 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:45.849147    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87ddf68f-44eb-473c-aa9d-785716cde3a6-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:28:45.849179 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:45.849176    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbwhd\" (UniqueName: \"kubernetes.io/projected/87ddf68f-44eb-473c-aa9d-785716cde3a6-kube-api-access-lbwhd\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:28:46.502281 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:46.502255    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b46dc8f6-t8tvg"
Nov 03 09:28:46.502545 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:46.502299    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b46dc8f6-t8tvg" event={"ID":"87ddf68f-44eb-473c-aa9d-785716cde3a6","Type":"ContainerDied","Data":"0a47e4c805421b697924b2aef2aa204088af7c389816bb6f496ed8c6e0499bd0"}
Nov 03 09:28:46.502545 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:46.502346    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a47e4c805421b697924b2aef2aa204088af7c389816bb6f496ed8c6e0499bd0"
Nov 03 09:28:46.954260 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:28:46.954220    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:28:52.993161 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:52.993129    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0371fedf-dnjwx"]
Nov 03 09:28:52.993645 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:52.993631    4473 cpu_manager.go:414] "containerMap: removing container" podUID="87ddf68f-44eb-473c-aa9d-785716cde3a6" containerName="revoke-keys"
Nov 03 09:28:52.993689 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:52.993646    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87ddf68f-44eb-473c-aa9d-785716cde3a6" containerName="revoke-keys"
Nov 03 09:28:52.993770 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:52.993761    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="87ddf68f-44eb-473c-aa9d-785716cde3a6" containerName="revoke-keys"
Nov 03 09:28:52.997114 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:52.997099    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0371fedf-dnjwx"
Nov 03 09:28:53.027521 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:53.027496    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b89d43eb-080a-4b2a-91d9-e712bf431043-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0371fedf-dnjwx\" (UID: \"b89d43eb-080a-4b2a-91d9-e712bf431043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0371fedf-dnjwx"
Nov 03 09:28:53.027648 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:53.027543    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rm8j\" (UniqueName: \"kubernetes.io/projected/b89d43eb-080a-4b2a-91d9-e712bf431043-kube-api-access-8rm8j\") pod \"maas-api-revoke-sub-0371fedf-dnjwx\" (UID: \"b89d43eb-080a-4b2a-91d9-e712bf431043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0371fedf-dnjwx"
Nov 03 09:28:53.027648 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:53.027569    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b89d43eb-080a-4b2a-91d9-e712bf431043-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0371fedf-dnjwx\" (UID: \"b89d43eb-080a-4b2a-91d9-e712bf431043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0371fedf-dnjwx"
Nov 03 09:28:53.055080 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:53.055053    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0371fedf-dnjwx"]
Nov 03 09:28:53.101493 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:53.101470    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:28:53.101493 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:53.101468    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:28:53.101676 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:53.101549    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:28:53.101676 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:53.101557    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:28:53.128617 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:53.128589    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rm8j\" (UniqueName: \"kubernetes.io/projected/b89d43eb-080a-4b2a-91d9-e712bf431043-kube-api-access-8rm8j\") pod \"maas-api-revoke-sub-0371fedf-dnjwx\" (UID: \"b89d43eb-080a-4b2a-91d9-e712bf431043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0371fedf-dnjwx"
Nov 03 09:28:53.128784 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:53.128624    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b89d43eb-080a-4b2a-91d9-e712bf431043-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0371fedf-dnjwx\" (UID: \"b89d43eb-080a-4b2a-91d9-e712bf431043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0371fedf-dnjwx"
Nov 03 09:28:53.128784 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:53.128747    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b89d43eb-080a-4b2a-91d9-e712bf431043-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0371fedf-dnjwx\" (UID: \"b89d43eb-080a-4b2a-91d9-e712bf431043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0371fedf-dnjwx"
Nov 03 09:28:53.129361 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:53.129343    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b89d43eb-080a-4b2a-91d9-e712bf431043-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0371fedf-dnjwx\" (UID: \"b89d43eb-080a-4b2a-91d9-e712bf431043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0371fedf-dnjwx"
Nov 03 09:28:53.131004 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:53.130986    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b89d43eb-080a-4b2a-91d9-e712bf431043-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0371fedf-dnjwx\" (UID: \"b89d43eb-080a-4b2a-91d9-e712bf431043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0371fedf-dnjwx"
Nov 03 09:28:53.188661 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:53.188623    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rm8j\" (UniqueName: \"kubernetes.io/projected/b89d43eb-080a-4b2a-91d9-e712bf431043-kube-api-access-8rm8j\") pod \"maas-api-revoke-sub-0371fedf-dnjwx\" (UID: \"b89d43eb-080a-4b2a-91d9-e712bf431043\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0371fedf-dnjwx"
Nov 03 09:28:53.310644 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:53.310559    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0371fedf-dnjwx"
Nov 03 09:28:53.687071 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:53.687042    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0371fedf-dnjwx"]
Nov 03 09:28:53.687431 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:28:53.687408    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb89d43eb_080a_4b2a_91d9_e712bf431043.slice/crio-53cc1792d6d07a736117bbcd577b96de75700df9194abbb27ed442f67376eb71 WatchSource:0}: Error finding container 53cc1792d6d07a736117bbcd577b96de75700df9194abbb27ed442f67376eb71: Status 404 returned error can't find the container with id 53cc1792d6d07a736117bbcd577b96de75700df9194abbb27ed442f67376eb71
Nov 03 09:28:54.551318 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:54.551281    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0371fedf-dnjwx" event={"ID":"b89d43eb-080a-4b2a-91d9-e712bf431043","Type":"ContainerStarted","Data":"769aaf5198e318896766b017e84819e3927b9a9b3faf1b34783dff4380611e86"}
Nov 03 09:28:54.551798 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:54.551338    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0371fedf-dnjwx" event={"ID":"b89d43eb-080a-4b2a-91d9-e712bf431043","Type":"ContainerStarted","Data":"53cc1792d6d07a736117bbcd577b96de75700df9194abbb27ed442f67376eb71"}
Nov 03 09:28:54.669532 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:54.669486    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0371fedf-dnjwx" podStartSLOduration=2.669473254 podStartE2EDuration="2.669473254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:28:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:28:54.666511624 +0000 UTC m=+2958.935575075" watchObservedRunningTime="2026-11-03 09:28:54.669473254 +0000 UTC m=+2958.938536687"
Nov 03 09:28:55.558819 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:55.558785    4473 generic.go:356] "Generic (PLEG): container finished" podID="b89d43eb-080a-4b2a-91d9-e712bf431043" containerID="769aaf5198e318896766b017e84819e3927b9a9b3faf1b34783dff4380611e86" exitCode=0
Nov 03 09:28:55.559225 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:55.558868    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0371fedf-dnjwx" event={"ID":"b89d43eb-080a-4b2a-91d9-e712bf431043","Type":"ContainerDied","Data":"769aaf5198e318896766b017e84819e3927b9a9b3faf1b34783dff4380611e86"}
Nov 03 09:28:56.716718 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:56.716695    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0371fedf-dnjwx"
Nov 03 09:28:56.769833 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:56.769806    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b89d43eb-080a-4b2a-91d9-e712bf431043-kube-api-access-8rm8j\" (UniqueName: \"kubernetes.io/projected/b89d43eb-080a-4b2a-91d9-e712bf431043-kube-api-access-8rm8j\") pod \"b89d43eb-080a-4b2a-91d9-e712bf431043\" (UID: \"b89d43eb-080a-4b2a-91d9-e712bf431043\") "
Nov 03 09:28:56.770016 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:56.769847    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b89d43eb-080a-4b2a-91d9-e712bf431043-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b89d43eb-080a-4b2a-91d9-e712bf431043-maas-api-openshift-ca\") pod \"b89d43eb-080a-4b2a-91d9-e712bf431043\" (UID: \"b89d43eb-080a-4b2a-91d9-e712bf431043\") "
Nov 03 09:28:56.770016 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:56.769878    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b89d43eb-080a-4b2a-91d9-e712bf431043-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b89d43eb-080a-4b2a-91d9-e712bf431043-maas-api-xks-ca\") pod \"b89d43eb-080a-4b2a-91d9-e712bf431043\" (UID: \"b89d43eb-080a-4b2a-91d9-e712bf431043\") "
Nov 03 09:28:56.770231 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:56.770211    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b89d43eb-080a-4b2a-91d9-e712bf431043-maas-api-openshift-ca" pod "b89d43eb-080a-4b2a-91d9-e712bf431043" (UID: "b89d43eb-080a-4b2a-91d9-e712bf431043"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:28:56.770398 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:56.770374    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b89d43eb-080a-4b2a-91d9-e712bf431043-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:28:56.772137 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:56.772109    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b89d43eb-080a-4b2a-91d9-e712bf431043-maas-api-xks-ca" pod "b89d43eb-080a-4b2a-91d9-e712bf431043" (UID: "b89d43eb-080a-4b2a-91d9-e712bf431043"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:28:56.772228 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:56.772181    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b89d43eb-080a-4b2a-91d9-e712bf431043-kube-api-access-8rm8j" pod "b89d43eb-080a-4b2a-91d9-e712bf431043" (UID: "b89d43eb-080a-4b2a-91d9-e712bf431043"). InnerVolumeSpecName "kube-api-access-8rm8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:28:56.871537 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:56.871511    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rm8j\" (UniqueName: \"kubernetes.io/projected/b89d43eb-080a-4b2a-91d9-e712bf431043-kube-api-access-8rm8j\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:28:56.871537 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:56.871538    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b89d43eb-080a-4b2a-91d9-e712bf431043-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:28:57.572194 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:57.572164    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0371fedf-dnjwx"
Nov 03 09:28:57.572366 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:57.572211    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0371fedf-dnjwx" event={"ID":"b89d43eb-080a-4b2a-91d9-e712bf431043","Type":"ContainerDied","Data":"53cc1792d6d07a736117bbcd577b96de75700df9194abbb27ed442f67376eb71"}
Nov 03 09:28:57.572366 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:28:57.572257    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53cc1792d6d07a736117bbcd577b96de75700df9194abbb27ed442f67376eb71"
Nov 03 09:29:03.157442 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:03.157411    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:29:06.734220 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:06.734180    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61722479-bp2bm"]
Nov 03 09:29:06.734985 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:06.734964    4473 cpu_manager.go:414] "containerMap: removing container" podUID="b89d43eb-080a-4b2a-91d9-e712bf431043" containerName="revoke-keys"
Nov 03 09:29:06.735083 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:06.734986    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b89d43eb-080a-4b2a-91d9-e712bf431043" containerName="revoke-keys"
Nov 03 09:29:06.735152 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:06.735137    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b89d43eb-080a-4b2a-91d9-e712bf431043" containerName="revoke-keys"
Nov 03 09:29:06.738754 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:06.738732    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61722479-bp2bm"
Nov 03 09:29:06.795347 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:06.795314    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61722479-bp2bm"]
Nov 03 09:29:06.830875 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:06.830841    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:29:06.841457 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:06.841427    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:29:06.841457 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:06.841456    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:29:06.841457 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:06.841467    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:29:06.880308 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:06.880274    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61722479-bp2bm\" (UID: \"bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61722479-bp2bm"
Nov 03 09:29:06.880469 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:06.880319    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61722479-bp2bm\" (UID: \"bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61722479-bp2bm"
Nov 03 09:29:06.880508 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:06.880472    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjblq\" (UniqueName: \"kubernetes.io/projected/bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8-kube-api-access-qjblq\") pod \"maas-api-revoke-sub-61722479-bp2bm\" (UID: \"bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61722479-bp2bm"
Nov 03 09:29:06.981584 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:06.981547    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61722479-bp2bm\" (UID: \"bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61722479-bp2bm"
Nov 03 09:29:06.981821 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:06.981784    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61722479-bp2bm\" (UID: \"bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61722479-bp2bm"
Nov 03 09:29:06.982015 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:06.981990    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjblq\" (UniqueName: \"kubernetes.io/projected/bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8-kube-api-access-qjblq\") pod \"maas-api-revoke-sub-61722479-bp2bm\" (UID: \"bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61722479-bp2bm"
Nov 03 09:29:06.982626 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:06.982606    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61722479-bp2bm\" (UID: \"bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61722479-bp2bm"
Nov 03 09:29:06.984289 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:06.984243    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61722479-bp2bm\" (UID: \"bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61722479-bp2bm"
Nov 03 09:29:07.044376 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:07.044344    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjblq\" (UniqueName: \"kubernetes.io/projected/bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8-kube-api-access-qjblq\") pod \"maas-api-revoke-sub-61722479-bp2bm\" (UID: \"bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61722479-bp2bm"
Nov 03 09:29:07.053025 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:07.052997    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61722479-bp2bm"
Nov 03 09:29:07.435825 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:07.435801    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61722479-bp2bm"]
Nov 03 09:29:07.437415 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:29:07.437381    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd1e4f82_6ff0_4b7d_9e23_9dd3779368c8.slice/crio-306d94829a6df5c73dd88d7d92413b7a23aaadbd1d717a74ed434ede894f1c21 WatchSource:0}: Error finding container 306d94829a6df5c73dd88d7d92413b7a23aaadbd1d717a74ed434ede894f1c21: Status 404 returned error can't find the container with id 306d94829a6df5c73dd88d7d92413b7a23aaadbd1d717a74ed434ede894f1c21
Nov 03 09:29:07.634672 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:07.634639    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61722479-bp2bm" event={"ID":"bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8","Type":"ContainerStarted","Data":"306d94829a6df5c73dd88d7d92413b7a23aaadbd1d717a74ed434ede894f1c21"}
Nov 03 09:29:08.642757 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:08.642724    4473 generic.go:356] "Generic (PLEG): container finished" podID="bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8" containerID="bdfd1224747d7cc1bb17bacca2aeeb6fa443b7e8875a9768ded19826a8e8348b" exitCode=0
Nov 03 09:29:08.643154 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:08.642802    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61722479-bp2bm" event={"ID":"bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8","Type":"ContainerDied","Data":"bdfd1224747d7cc1bb17bacca2aeeb6fa443b7e8875a9768ded19826a8e8348b"}
Nov 03 09:29:09.248360 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:09.248325    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-786f48c9-lvxgh"]
Nov 03 09:29:09.252239 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:09.252219    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786f48c9-lvxgh"
Nov 03 09:29:09.310679 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:09.310582    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-786f48c9-lvxgh"]
Nov 03 09:29:09.408268 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:09.408231    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp8wv\" (UniqueName: \"kubernetes.io/projected/74da5b90-e2f4-4a11-82fd-ce5fb88a068c-kube-api-access-qp8wv\") pod \"maas-api-revoke-sub-786f48c9-lvxgh\" (UID: \"74da5b90-e2f4-4a11-82fd-ce5fb88a068c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786f48c9-lvxgh"
Nov 03 09:29:09.408449 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:09.408360    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74da5b90-e2f4-4a11-82fd-ce5fb88a068c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-786f48c9-lvxgh\" (UID: \"74da5b90-e2f4-4a11-82fd-ce5fb88a068c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786f48c9-lvxgh"
Nov 03 09:29:09.408449 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:09.408428    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74da5b90-e2f4-4a11-82fd-ce5fb88a068c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-786f48c9-lvxgh\" (UID: \"74da5b90-e2f4-4a11-82fd-ce5fb88a068c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786f48c9-lvxgh"
Nov 03 09:29:09.509759 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:09.509681    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74da5b90-e2f4-4a11-82fd-ce5fb88a068c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-786f48c9-lvxgh\" (UID: \"74da5b90-e2f4-4a11-82fd-ce5fb88a068c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786f48c9-lvxgh"
Nov 03 09:29:09.509934 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:09.509781    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp8wv\" (UniqueName: \"kubernetes.io/projected/74da5b90-e2f4-4a11-82fd-ce5fb88a068c-kube-api-access-qp8wv\") pod \"maas-api-revoke-sub-786f48c9-lvxgh\" (UID: \"74da5b90-e2f4-4a11-82fd-ce5fb88a068c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786f48c9-lvxgh"
Nov 03 09:29:09.509934 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:09.509825    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74da5b90-e2f4-4a11-82fd-ce5fb88a068c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-786f48c9-lvxgh\" (UID: \"74da5b90-e2f4-4a11-82fd-ce5fb88a068c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786f48c9-lvxgh"
Nov 03 09:29:09.510490 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:09.510472    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74da5b90-e2f4-4a11-82fd-ce5fb88a068c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-786f48c9-lvxgh\" (UID: \"74da5b90-e2f4-4a11-82fd-ce5fb88a068c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786f48c9-lvxgh"
Nov 03 09:29:09.512171 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:09.512152    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74da5b90-e2f4-4a11-82fd-ce5fb88a068c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-786f48c9-lvxgh\" (UID: \"74da5b90-e2f4-4a11-82fd-ce5fb88a068c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786f48c9-lvxgh"
Nov 03 09:29:09.570191 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:09.570160    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp8wv\" (UniqueName: \"kubernetes.io/projected/74da5b90-e2f4-4a11-82fd-ce5fb88a068c-kube-api-access-qp8wv\") pod \"maas-api-revoke-sub-786f48c9-lvxgh\" (UID: \"74da5b90-e2f4-4a11-82fd-ce5fb88a068c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-786f48c9-lvxgh"
Nov 03 09:29:09.803541 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:09.803518    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61722479-bp2bm"
Nov 03 09:29:09.865010 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:09.864982    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786f48c9-lvxgh"
Nov 03 09:29:09.913595 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:09.913564    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8-kube-api-access-qjblq\" (UniqueName: \"kubernetes.io/projected/bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8-kube-api-access-qjblq\") pod \"bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8\" (UID: \"bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8\") "
Nov 03 09:29:09.913794 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:09.913638    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8-maas-api-xks-ca\") pod \"bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8\" (UID: \"bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8\") "
Nov 03 09:29:09.913794 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:09.913736    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8-maas-api-openshift-ca\") pod \"bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8\" (UID: \"bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8\") "
Nov 03 09:29:09.914280 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:09.914243    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8-maas-api-openshift-ca" pod "bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8" (UID: "bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:29:09.914692 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:09.914554    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:29:09.916578 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:09.916550    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8-kube-api-access-qjblq" pod "bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8" (UID: "bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8"). InnerVolumeSpecName "kube-api-access-qjblq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:29:09.916705 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:09.916599    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8-maas-api-xks-ca" pod "bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8" (UID: "bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:29:10.015867 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:10.015836    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjblq\" (UniqueName: \"kubernetes.io/projected/bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8-kube-api-access-qjblq\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:29:10.015867 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:10.015864    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:29:10.247257 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:10.247227    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-786f48c9-lvxgh"]
Nov 03 09:29:10.248242 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:29:10.248219    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74da5b90_e2f4_4a11_82fd_ce5fb88a068c.slice/crio-f4b5c31bc30d2d5c13edc69fa1002c3642fd74422f4dc4ee6f99422304de6274 WatchSource:0}: Error finding container f4b5c31bc30d2d5c13edc69fa1002c3642fd74422f4dc4ee6f99422304de6274: Status 404 returned error can't find the container with id f4b5c31bc30d2d5c13edc69fa1002c3642fd74422f4dc4ee6f99422304de6274
Nov 03 09:29:10.657201 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:10.657148    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786f48c9-lvxgh" event={"ID":"74da5b90-e2f4-4a11-82fd-ce5fb88a068c","Type":"ContainerStarted","Data":"902a9af4b01f6b369ade740aa06bfc85ae7dda0fb964c67f067e4fe4f6044aa6"}
Nov 03 09:29:10.657499 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:10.657481    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786f48c9-lvxgh" event={"ID":"74da5b90-e2f4-4a11-82fd-ce5fb88a068c","Type":"ContainerStarted","Data":"f4b5c31bc30d2d5c13edc69fa1002c3642fd74422f4dc4ee6f99422304de6274"}
Nov 03 09:29:10.660742 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:10.660703    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61722479-bp2bm" event={"ID":"bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8","Type":"ContainerDied","Data":"306d94829a6df5c73dd88d7d92413b7a23aaadbd1d717a74ed434ede894f1c21"}
Nov 03 09:29:10.660868 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:10.660753    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="306d94829a6df5c73dd88d7d92413b7a23aaadbd1d717a74ed434ede894f1c21"
Nov 03 09:29:10.660868 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:10.660827    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61722479-bp2bm"
Nov 03 09:29:10.775569 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:10.775461    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786f48c9-lvxgh" podStartSLOduration=1.775445093 podStartE2EDuration="1.775445093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:29:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:29:10.773088079 +0000 UTC m=+2975.042151532" watchObservedRunningTime="2026-11-03 09:29:10.775445093 +0000 UTC m=+2975.044508528"
Nov 03 09:29:11.668518 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:11.668486    4473 generic.go:356] "Generic (PLEG): container finished" podID="74da5b90-e2f4-4a11-82fd-ce5fb88a068c" containerID="902a9af4b01f6b369ade740aa06bfc85ae7dda0fb964c67f067e4fe4f6044aa6" exitCode=0
Nov 03 09:29:11.668907 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:11.668567    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786f48c9-lvxgh" event={"ID":"74da5b90-e2f4-4a11-82fd-ce5fb88a068c","Type":"ContainerDied","Data":"902a9af4b01f6b369ade740aa06bfc85ae7dda0fb964c67f067e4fe4f6044aa6"}
Nov 03 09:29:12.817799 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:12.817773    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786f48c9-lvxgh"
Nov 03 09:29:12.947682 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:12.947599    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74da5b90-e2f4-4a11-82fd-ce5fb88a068c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74da5b90-e2f4-4a11-82fd-ce5fb88a068c-maas-api-xks-ca\") pod \"74da5b90-e2f4-4a11-82fd-ce5fb88a068c\" (UID: \"74da5b90-e2f4-4a11-82fd-ce5fb88a068c\") "
Nov 03 09:29:12.947839 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:12.947790    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74da5b90-e2f4-4a11-82fd-ce5fb88a068c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74da5b90-e2f4-4a11-82fd-ce5fb88a068c-maas-api-openshift-ca\") pod \"74da5b90-e2f4-4a11-82fd-ce5fb88a068c\" (UID: \"74da5b90-e2f4-4a11-82fd-ce5fb88a068c\") "
Nov 03 09:29:12.947976 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:12.947925    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74da5b90-e2f4-4a11-82fd-ce5fb88a068c-kube-api-access-qp8wv\" (UniqueName: \"kubernetes.io/projected/74da5b90-e2f4-4a11-82fd-ce5fb88a068c-kube-api-access-qp8wv\") pod \"74da5b90-e2f4-4a11-82fd-ce5fb88a068c\" (UID: \"74da5b90-e2f4-4a11-82fd-ce5fb88a068c\") "
Nov 03 09:29:12.948183 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:12.948161    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74da5b90-e2f4-4a11-82fd-ce5fb88a068c-maas-api-openshift-ca" pod "74da5b90-e2f4-4a11-82fd-ce5fb88a068c" (UID: "74da5b90-e2f4-4a11-82fd-ce5fb88a068c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:29:12.948625 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:12.948601    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74da5b90-e2f4-4a11-82fd-ce5fb88a068c-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:29:12.950074 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:12.950048    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74da5b90-e2f4-4a11-82fd-ce5fb88a068c-maas-api-xks-ca" pod "74da5b90-e2f4-4a11-82fd-ce5fb88a068c" (UID: "74da5b90-e2f4-4a11-82fd-ce5fb88a068c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:29:12.950206 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:12.950183    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74da5b90-e2f4-4a11-82fd-ce5fb88a068c-kube-api-access-qp8wv" pod "74da5b90-e2f4-4a11-82fd-ce5fb88a068c" (UID: "74da5b90-e2f4-4a11-82fd-ce5fb88a068c"). InnerVolumeSpecName "kube-api-access-qp8wv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:29:13.049737 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:13.049698    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp8wv\" (UniqueName: \"kubernetes.io/projected/74da5b90-e2f4-4a11-82fd-ce5fb88a068c-kube-api-access-qp8wv\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:29:13.049737 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:13.049730    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74da5b90-e2f4-4a11-82fd-ce5fb88a068c-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:29:13.681322 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:13.681290    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786f48c9-lvxgh" event={"ID":"74da5b90-e2f4-4a11-82fd-ce5fb88a068c","Type":"ContainerDied","Data":"f4b5c31bc30d2d5c13edc69fa1002c3642fd74422f4dc4ee6f99422304de6274"}
Nov 03 09:29:13.681322 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:13.681316    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-786f48c9-lvxgh"
Nov 03 09:29:13.681544 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:13.681343    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4b5c31bc30d2d5c13edc69fa1002c3642fd74422f4dc4ee6f99422304de6274"
Nov 03 09:29:16.938832 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:29:16.938792    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:29:17.488373 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:17.488335    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80bbfbc3-hhgbz"]
Nov 03 09:29:17.488904 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:17.488889    4473 cpu_manager.go:414] "containerMap: removing container" podUID="bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8" containerName="revoke-keys"
Nov 03 09:29:17.488968 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:17.488905    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8" containerName="revoke-keys"
Nov 03 09:29:17.488968 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:17.488922    4473 cpu_manager.go:414] "containerMap: removing container" podUID="74da5b90-e2f4-4a11-82fd-ce5fb88a068c" containerName="revoke-keys"
Nov 03 09:29:17.488968 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:17.488928    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74da5b90-e2f4-4a11-82fd-ce5fb88a068c" containerName="revoke-keys"
Nov 03 09:29:17.489071 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:17.489014    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="74da5b90-e2f4-4a11-82fd-ce5fb88a068c" containerName="revoke-keys"
Nov 03 09:29:17.489071 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:17.489032    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8" containerName="revoke-keys"
Nov 03 09:29:17.492124 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:17.492103    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80bbfbc3-hhgbz"
Nov 03 09:29:17.503420 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:17.503402    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69becb1c-279e-4ef0-b187-ae7f60df3d43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80bbfbc3-hhgbz\" (UID: \"69becb1c-279e-4ef0-b187-ae7f60df3d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80bbfbc3-hhgbz"
Nov 03 09:29:17.503502 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:17.503448    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69becb1c-279e-4ef0-b187-ae7f60df3d43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80bbfbc3-hhgbz\" (UID: \"69becb1c-279e-4ef0-b187-ae7f60df3d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80bbfbc3-hhgbz"
Nov 03 09:29:17.503577 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:17.503557    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prr5z\" (UniqueName: \"kubernetes.io/projected/69becb1c-279e-4ef0-b187-ae7f60df3d43-kube-api-access-prr5z\") pod \"maas-api-revoke-sub-80bbfbc3-hhgbz\" (UID: \"69becb1c-279e-4ef0-b187-ae7f60df3d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80bbfbc3-hhgbz"
Nov 03 09:29:17.550730 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:17.550698    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80bbfbc3-hhgbz"]
Nov 03 09:29:17.596532 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:17.596504    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:29:17.596532 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:17.596524    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:29:17.596754 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:17.596504    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:29:17.596754 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:17.596570    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:29:17.604531 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:17.604504    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69becb1c-279e-4ef0-b187-ae7f60df3d43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80bbfbc3-hhgbz\" (UID: \"69becb1c-279e-4ef0-b187-ae7f60df3d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80bbfbc3-hhgbz"
Nov 03 09:29:17.604651 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:17.604582    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prr5z\" (UniqueName: \"kubernetes.io/projected/69becb1c-279e-4ef0-b187-ae7f60df3d43-kube-api-access-prr5z\") pod \"maas-api-revoke-sub-80bbfbc3-hhgbz\" (UID: \"69becb1c-279e-4ef0-b187-ae7f60df3d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80bbfbc3-hhgbz"
Nov 03 09:29:17.604745 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:17.604728    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69becb1c-279e-4ef0-b187-ae7f60df3d43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80bbfbc3-hhgbz\" (UID: \"69becb1c-279e-4ef0-b187-ae7f60df3d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80bbfbc3-hhgbz"
Nov 03 09:29:17.605232 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:17.605209    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69becb1c-279e-4ef0-b187-ae7f60df3d43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80bbfbc3-hhgbz\" (UID: \"69becb1c-279e-4ef0-b187-ae7f60df3d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80bbfbc3-hhgbz"
Nov 03 09:29:17.607187 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:17.607169    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69becb1c-279e-4ef0-b187-ae7f60df3d43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80bbfbc3-hhgbz\" (UID: \"69becb1c-279e-4ef0-b187-ae7f60df3d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80bbfbc3-hhgbz"
Nov 03 09:29:17.665310 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:17.665284    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prr5z\" (UniqueName: \"kubernetes.io/projected/69becb1c-279e-4ef0-b187-ae7f60df3d43-kube-api-access-prr5z\") pod \"maas-api-revoke-sub-80bbfbc3-hhgbz\" (UID: \"69becb1c-279e-4ef0-b187-ae7f60df3d43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80bbfbc3-hhgbz"
Nov 03 09:29:17.805497 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:17.805429    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80bbfbc3-hhgbz"
Nov 03 09:29:18.186870 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:18.186845    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80bbfbc3-hhgbz"]
Nov 03 09:29:18.188164 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:29:18.188138    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69becb1c_279e_4ef0_b187_ae7f60df3d43.slice/crio-3a6d29e980f914bc5dc577ac1b8b90a29d8f0d25a6eaa786c594e94582a8f2cd WatchSource:0}: Error finding container 3a6d29e980f914bc5dc577ac1b8b90a29d8f0d25a6eaa786c594e94582a8f2cd: Status 404 returned error can't find the container with id 3a6d29e980f914bc5dc577ac1b8b90a29d8f0d25a6eaa786c594e94582a8f2cd
Nov 03 09:29:18.711278 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:18.711240    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80bbfbc3-hhgbz" event={"ID":"69becb1c-279e-4ef0-b187-ae7f60df3d43","Type":"ContainerStarted","Data":"436b9af9de4373f07a3e26bce7fef15a12df1d39203b71609f2e35d3746d46da"}
Nov 03 09:29:18.711452 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:18.711296    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80bbfbc3-hhgbz" event={"ID":"69becb1c-279e-4ef0-b187-ae7f60df3d43","Type":"ContainerStarted","Data":"3a6d29e980f914bc5dc577ac1b8b90a29d8f0d25a6eaa786c594e94582a8f2cd"}
Nov 03 09:29:18.828725 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:18.828615    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80bbfbc3-hhgbz" podStartSLOduration=1.82860098 podStartE2EDuration="1.82860098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:29:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:29:18.826430049 +0000 UTC m=+2983.095493496" watchObservedRunningTime="2026-11-03 09:29:18.82860098 +0000 UTC m=+2983.097664414"
Nov 03 09:29:19.724095 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:19.724058    4473 generic.go:356] "Generic (PLEG): container finished" podID="69becb1c-279e-4ef0-b187-ae7f60df3d43" containerID="436b9af9de4373f07a3e26bce7fef15a12df1d39203b71609f2e35d3746d46da" exitCode=0
Nov 03 09:29:19.724590 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:19.724135    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80bbfbc3-hhgbz" event={"ID":"69becb1c-279e-4ef0-b187-ae7f60df3d43","Type":"ContainerDied","Data":"436b9af9de4373f07a3e26bce7fef15a12df1d39203b71609f2e35d3746d46da"}
Nov 03 09:29:20.003922 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:20.003851    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bb21c52-cmgl4"]
Nov 03 09:29:20.007583 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:20.007561    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb21c52-cmgl4"
Nov 03 09:29:20.037730 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:20.037705    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42779912-fcdc-45e4-bcbf-65e4ae3e3a27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bb21c52-cmgl4\" (UID: \"42779912-fcdc-45e4-bcbf-65e4ae3e3a27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb21c52-cmgl4"
Nov 03 09:29:20.037854 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:20.037757    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st7qr\" (UniqueName: \"kubernetes.io/projected/42779912-fcdc-45e4-bcbf-65e4ae3e3a27-kube-api-access-st7qr\") pod \"maas-api-revoke-sub-0bb21c52-cmgl4\" (UID: \"42779912-fcdc-45e4-bcbf-65e4ae3e3a27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb21c52-cmgl4"
Nov 03 09:29:20.037925 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:20.037907    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42779912-fcdc-45e4-bcbf-65e4ae3e3a27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bb21c52-cmgl4\" (UID: \"42779912-fcdc-45e4-bcbf-65e4ae3e3a27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb21c52-cmgl4"
Nov 03 09:29:20.070488 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:20.070456    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bb21c52-cmgl4"]
Nov 03 09:29:20.138770 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:20.138734    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42779912-fcdc-45e4-bcbf-65e4ae3e3a27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bb21c52-cmgl4\" (UID: \"42779912-fcdc-45e4-bcbf-65e4ae3e3a27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb21c52-cmgl4"
Nov 03 09:29:20.138962 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:20.138824    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st7qr\" (UniqueName: \"kubernetes.io/projected/42779912-fcdc-45e4-bcbf-65e4ae3e3a27-kube-api-access-st7qr\") pod \"maas-api-revoke-sub-0bb21c52-cmgl4\" (UID: \"42779912-fcdc-45e4-bcbf-65e4ae3e3a27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb21c52-cmgl4"
Nov 03 09:29:20.138962 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:20.138903    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42779912-fcdc-45e4-bcbf-65e4ae3e3a27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bb21c52-cmgl4\" (UID: \"42779912-fcdc-45e4-bcbf-65e4ae3e3a27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb21c52-cmgl4"
Nov 03 09:29:20.139593 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:20.139572    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42779912-fcdc-45e4-bcbf-65e4ae3e3a27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bb21c52-cmgl4\" (UID: \"42779912-fcdc-45e4-bcbf-65e4ae3e3a27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb21c52-cmgl4"
Nov 03 09:29:20.141294 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:20.141270    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42779912-fcdc-45e4-bcbf-65e4ae3e3a27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bb21c52-cmgl4\" (UID: \"42779912-fcdc-45e4-bcbf-65e4ae3e3a27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb21c52-cmgl4"
Nov 03 09:29:20.199960 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:20.199913    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st7qr\" (UniqueName: \"kubernetes.io/projected/42779912-fcdc-45e4-bcbf-65e4ae3e3a27-kube-api-access-st7qr\") pod \"maas-api-revoke-sub-0bb21c52-cmgl4\" (UID: \"42779912-fcdc-45e4-bcbf-65e4ae3e3a27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb21c52-cmgl4"
Nov 03 09:29:20.321491 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:20.321413    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb21c52-cmgl4"
Nov 03 09:29:20.698611 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:20.698589    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bb21c52-cmgl4"]
Nov 03 09:29:20.734778 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:20.734752    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb21c52-cmgl4" event={"ID":"42779912-fcdc-45e4-bcbf-65e4ae3e3a27","Type":"ContainerStarted","Data":"88c4bb6401dc1587df36c9275067ac384d603a2a136b275e97b5fb10ad7fd773"}
Nov 03 09:29:20.931371 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:20.931346    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80bbfbc3-hhgbz"
Nov 03 09:29:20.951045 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:20.950978    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69becb1c-279e-4ef0-b187-ae7f60df3d43-kube-api-access-prr5z\" (UniqueName: \"kubernetes.io/projected/69becb1c-279e-4ef0-b187-ae7f60df3d43-kube-api-access-prr5z\") pod \"69becb1c-279e-4ef0-b187-ae7f60df3d43\" (UID: \"69becb1c-279e-4ef0-b187-ae7f60df3d43\") "
Nov 03 09:29:20.951045 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:20.951016    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69becb1c-279e-4ef0-b187-ae7f60df3d43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69becb1c-279e-4ef0-b187-ae7f60df3d43-maas-api-xks-ca\") pod \"69becb1c-279e-4ef0-b187-ae7f60df3d43\" (UID: \"69becb1c-279e-4ef0-b187-ae7f60df3d43\") "
Nov 03 09:29:20.951249 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:20.951093    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69becb1c-279e-4ef0-b187-ae7f60df3d43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69becb1c-279e-4ef0-b187-ae7f60df3d43-maas-api-openshift-ca\") pod \"69becb1c-279e-4ef0-b187-ae7f60df3d43\" (UID: \"69becb1c-279e-4ef0-b187-ae7f60df3d43\") "
Nov 03 09:29:20.951522 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:20.951494    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69becb1c-279e-4ef0-b187-ae7f60df3d43-maas-api-openshift-ca" pod "69becb1c-279e-4ef0-b187-ae7f60df3d43" (UID: "69becb1c-279e-4ef0-b187-ae7f60df3d43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:29:20.953425 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:20.953403    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69becb1c-279e-4ef0-b187-ae7f60df3d43-maas-api-xks-ca" pod "69becb1c-279e-4ef0-b187-ae7f60df3d43" (UID: "69becb1c-279e-4ef0-b187-ae7f60df3d43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:29:20.953425 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:20.953417    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69becb1c-279e-4ef0-b187-ae7f60df3d43-kube-api-access-prr5z" pod "69becb1c-279e-4ef0-b187-ae7f60df3d43" (UID: "69becb1c-279e-4ef0-b187-ae7f60df3d43"). InnerVolumeSpecName "kube-api-access-prr5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:29:21.053035 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:21.053009    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prr5z\" (UniqueName: \"kubernetes.io/projected/69becb1c-279e-4ef0-b187-ae7f60df3d43-kube-api-access-prr5z\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:29:21.053185 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:21.053039    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69becb1c-279e-4ef0-b187-ae7f60df3d43-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:29:21.053185 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:21.053053    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69becb1c-279e-4ef0-b187-ae7f60df3d43-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:29:21.742925 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:21.742890    4473 generic.go:356] "Generic (PLEG): container finished" podID="42779912-fcdc-45e4-bcbf-65e4ae3e3a27" containerID="78451d534563c58ccec8763542b38417f9d480c261030512472794901f1c83d5" exitCode=0
Nov 03 09:29:21.743356 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:21.742983    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb21c52-cmgl4" event={"ID":"42779912-fcdc-45e4-bcbf-65e4ae3e3a27","Type":"ContainerDied","Data":"78451d534563c58ccec8763542b38417f9d480c261030512472794901f1c83d5"}
Nov 03 09:29:21.744356 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:21.744334    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80bbfbc3-hhgbz" event={"ID":"69becb1c-279e-4ef0-b187-ae7f60df3d43","Type":"ContainerDied","Data":"3a6d29e980f914bc5dc577ac1b8b90a29d8f0d25a6eaa786c594e94582a8f2cd"}
Nov 03 09:29:21.744479 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:21.744384    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a6d29e980f914bc5dc577ac1b8b90a29d8f0d25a6eaa786c594e94582a8f2cd"
Nov 03 09:29:21.744479 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:21.744340    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80bbfbc3-hhgbz"
Nov 03 09:29:22.888039 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:22.888014    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb21c52-cmgl4"
Nov 03 09:29:22.974725 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:22.974693    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42779912-fcdc-45e4-bcbf-65e4ae3e3a27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42779912-fcdc-45e4-bcbf-65e4ae3e3a27-maas-api-openshift-ca\") pod \"42779912-fcdc-45e4-bcbf-65e4ae3e3a27\" (UID: \"42779912-fcdc-45e4-bcbf-65e4ae3e3a27\") "
Nov 03 09:29:22.974910 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:22.974741    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42779912-fcdc-45e4-bcbf-65e4ae3e3a27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42779912-fcdc-45e4-bcbf-65e4ae3e3a27-maas-api-xks-ca\") pod \"42779912-fcdc-45e4-bcbf-65e4ae3e3a27\" (UID: \"42779912-fcdc-45e4-bcbf-65e4ae3e3a27\") "
Nov 03 09:29:22.975057 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:22.975042    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42779912-fcdc-45e4-bcbf-65e4ae3e3a27-kube-api-access-st7qr\" (UniqueName: \"kubernetes.io/projected/42779912-fcdc-45e4-bcbf-65e4ae3e3a27-kube-api-access-st7qr\") pod \"42779912-fcdc-45e4-bcbf-65e4ae3e3a27\" (UID: \"42779912-fcdc-45e4-bcbf-65e4ae3e3a27\") "
Nov 03 09:29:22.975167 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:22.975144    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42779912-fcdc-45e4-bcbf-65e4ae3e3a27-maas-api-openshift-ca" pod "42779912-fcdc-45e4-bcbf-65e4ae3e3a27" (UID: "42779912-fcdc-45e4-bcbf-65e4ae3e3a27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:29:22.975432 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:22.975414    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42779912-fcdc-45e4-bcbf-65e4ae3e3a27-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:29:22.977086 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:22.977058    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42779912-fcdc-45e4-bcbf-65e4ae3e3a27-maas-api-xks-ca" pod "42779912-fcdc-45e4-bcbf-65e4ae3e3a27" (UID: "42779912-fcdc-45e4-bcbf-65e4ae3e3a27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:29:22.977233 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:22.977211    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42779912-fcdc-45e4-bcbf-65e4ae3e3a27-kube-api-access-st7qr" pod "42779912-fcdc-45e4-bcbf-65e4ae3e3a27" (UID: "42779912-fcdc-45e4-bcbf-65e4ae3e3a27"). InnerVolumeSpecName "kube-api-access-st7qr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:29:23.076908 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:23.076818    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st7qr\" (UniqueName: \"kubernetes.io/projected/42779912-fcdc-45e4-bcbf-65e4ae3e3a27-kube-api-access-st7qr\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:29:23.076908 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:23.076850    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42779912-fcdc-45e4-bcbf-65e4ae3e3a27-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:29:23.757994 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:23.757938    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb21c52-cmgl4"
Nov 03 09:29:23.758184 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:23.757987    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bb21c52-cmgl4" event={"ID":"42779912-fcdc-45e4-bcbf-65e4ae3e3a27","Type":"ContainerDied","Data":"88c4bb6401dc1587df36c9275067ac384d603a2a136b275e97b5fb10ad7fd773"}
Nov 03 09:29:23.758184 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:23.758045    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88c4bb6401dc1587df36c9275067ac384d603a2a136b275e97b5fb10ad7fd773"
Nov 03 09:29:36.446502 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:36.446468    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 09:29:36.453312 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:36.453291    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 09:29:46.937060 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:29:46.936986    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:29:56.229442 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:56.229403    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:29:58.878812 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:29:58.878781    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-433821/auth-test-model-433821-kserve-b9dc66df-7psmx"]
Nov 03 09:30:00.204001 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.203962    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894970-swn74"]
Nov 03 09:30:00.204793 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.204766    4473 cpu_manager.go:414] "containerMap: removing container" podUID="69becb1c-279e-4ef0-b187-ae7f60df3d43" containerName="revoke-keys"
Nov 03 09:30:00.204793 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.204788    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69becb1c-279e-4ef0-b187-ae7f60df3d43" containerName="revoke-keys"
Nov 03 09:30:00.204992 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.204814    4473 cpu_manager.go:414] "containerMap: removing container" podUID="42779912-fcdc-45e4-bcbf-65e4ae3e3a27" containerName="revoke-keys"
Nov 03 09:30:00.204992 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.204819    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42779912-fcdc-45e4-bcbf-65e4ae3e3a27" containerName="revoke-keys"
Nov 03 09:30:00.204992 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.204905    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="69becb1c-279e-4ef0-b187-ae7f60df3d43" containerName="revoke-keys"
Nov 03 09:30:00.204992 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.204918    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="42779912-fcdc-45e4-bcbf-65e4ae3e3a27" containerName="revoke-keys"
Nov 03 09:30:00.209269 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.209243    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894970-27x62"]
Nov 03 09:30:00.209409 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.209388    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894970-swn74"
Nov 03 09:30:00.212879 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.212857    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894970-4zzr8"]
Nov 03 09:30:00.213010 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.212982    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894970-27x62"
Nov 03 09:30:00.216472 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.216449    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894970-4zzr8"
Nov 03 09:30:00.254600 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.254564    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894970-xq4mw"]
Nov 03 09:30:00.258501 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.258479    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894970-4g892"]
Nov 03 09:30:00.258654 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.258601    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894970-xq4mw"
Nov 03 09:30:00.263235 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.263216    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894970-qgxzf"]
Nov 03 09:30:00.263347 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.263327    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894970-4g892"
Nov 03 09:30:00.267372 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.267266    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894970-lkt9m"]
Nov 03 09:30:00.267462 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.267393    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894970-qgxzf"
Nov 03 09:30:00.271718 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.271697    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894970-swn74"]
Nov 03 09:30:00.271838 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.271791    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894970-lkt9m"
Nov 03 09:30:00.301455 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.301427    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7slg4\" (UniqueName: \"kubernetes.io/projected/475138bd-0332-4a16-b413-272ccf954d5a-kube-api-access-7slg4\") pod \"maas-api-key-cleanup-e2e-worker-w0-910d50-29894970-xq4mw\" (UID: \"475138bd-0332-4a16-b413-272ccf954d5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894970-xq4mw"
Nov 03 09:30:00.301600 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.301563    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e5001f-f741-4337-8767-8297949ac190-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-433821-29894970-4zzr8\" (UID: \"68e5001f-f741-4337-8767-8297949ac190\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894970-4zzr8"
Nov 03 09:30:00.301672 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.301602    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccf23ce8-844c-4924-bd1c-e90e1ea762ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-465707-29894970-4g892\" (UID: \"ccf23ce8-844c-4924-bd1c-e90e1ea762ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894970-4g892"
Nov 03 09:30:00.301672 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.301628    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fg8r\" (UniqueName: \"kubernetes.io/projected/ccf23ce8-844c-4924-bd1c-e90e1ea762ab-kube-api-access-2fg8r\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-465707-29894970-4g892\" (UID: \"ccf23ce8-844c-4924-bd1c-e90e1ea762ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894970-4g892"
Nov 03 09:30:00.301672 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.301665    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/475138bd-0332-4a16-b413-272ccf954d5a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-910d50-29894970-xq4mw\" (UID: \"475138bd-0332-4a16-b413-272ccf954d5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894970-xq4mw"
Nov 03 09:30:00.301780 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.301745    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwq44\" (UniqueName: \"kubernetes.io/projected/5f055424-32fe-49ae-b6a9-4243eaf0c50b-kube-api-access-bwq44\") pod \"maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894970-27x62\" (UID: \"5f055424-32fe-49ae-b6a9-4243eaf0c50b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894970-27x62"
Nov 03 09:30:00.301780 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.301775    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phbs2\" (UniqueName: \"kubernetes.io/projected/68e5001f-f741-4337-8767-8297949ac190-kube-api-access-phbs2\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-433821-29894970-4zzr8\" (UID: \"68e5001f-f741-4337-8767-8297949ac190\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894970-4zzr8"
Nov 03 09:30:00.301898 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.301875    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6c7f\" (UniqueName: \"kubernetes.io/projected/b18d69e0-049e-4acd-bb6c-59299939e791-kube-api-access-r6c7f\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894970-swn74\" (UID: \"b18d69e0-049e-4acd-bb6c-59299939e791\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894970-swn74"
Nov 03 09:30:00.301974 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.301928    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b18d69e0-049e-4acd-bb6c-59299939e791-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894970-swn74\" (UID: \"b18d69e0-049e-4acd-bb6c-59299939e791\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894970-swn74"
Nov 03 09:30:00.302031 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.301997    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a436ee-2304-4a63-9cd9-82e65871c169-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894970-qgxzf\" (UID: \"b1a436ee-2304-4a63-9cd9-82e65871c169\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894970-qgxzf"
Nov 03 09:30:00.302125 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.302098    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b9x5\" (UniqueName: \"kubernetes.io/projected/b1a436ee-2304-4a63-9cd9-82e65871c169-kube-api-access-6b9x5\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894970-qgxzf\" (UID: \"b1a436ee-2304-4a63-9cd9-82e65871c169\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894970-qgxzf"
Nov 03 09:30:00.302202 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.302185    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f055424-32fe-49ae-b6a9-4243eaf0c50b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894970-27x62\" (UID: \"5f055424-32fe-49ae-b6a9-4243eaf0c50b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894970-27x62"
Nov 03 09:30:00.302268 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.302198    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:30:00.302268 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.302190    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:30:00.302268 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.302252    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:30:00.306754 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.306730    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894970-f5nvf"]
Nov 03 09:30:00.314965 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.311869    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894970-f5nvf"
Nov 03 09:30:00.318965 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.316755    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894970-l595n"]
Nov 03 09:30:00.330111 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.330084    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894970-xq4mw"]
Nov 03 09:30:00.330252 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.330136    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894970-4zzr8"]
Nov 03 09:30:00.330252 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.330199    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894970-l595n"
Nov 03 09:30:00.333868 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.333841    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894970-qgxzf"]
Nov 03 09:30:00.338504 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.338481    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894970-lkt9m"]
Nov 03 09:30:00.361017 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.360986    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894970-f5nvf"]
Nov 03 09:30:00.364871 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.364369    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894970-l595n"]
Nov 03 09:30:00.368719 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.368673    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894970-27x62"]
Nov 03 09:30:00.372078 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.372050    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894970-4g892"]
Nov 03 09:30:00.402983 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.402935    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b18d69e0-049e-4acd-bb6c-59299939e791-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894970-swn74\" (UID: \"b18d69e0-049e-4acd-bb6c-59299939e791\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894970-swn74"
Nov 03 09:30:00.403228 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.403003    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc8k5\" (UniqueName: \"kubernetes.io/projected/7f59e915-c5e9-4ca4-b67f-fcdec7f83136-kube-api-access-bc8k5\") pod \"maas-api-key-cleanup-29894970-lkt9m\" (UID: \"7f59e915-c5e9-4ca4-b67f-fcdec7f83136\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894970-lkt9m"
Nov 03 09:30:00.403228 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.403025    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a436ee-2304-4a63-9cd9-82e65871c169-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894970-qgxzf\" (UID: \"b1a436ee-2304-4a63-9cd9-82e65871c169\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894970-qgxzf"
Nov 03 09:30:00.403228 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.403050    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f59e915-c5e9-4ca4-b67f-fcdec7f83136-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894970-lkt9m\" (UID: \"7f59e915-c5e9-4ca4-b67f-fcdec7f83136\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894970-lkt9m"
Nov 03 09:30:00.403228 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.403073    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b9x5\" (UniqueName: \"kubernetes.io/projected/b1a436ee-2304-4a63-9cd9-82e65871c169-kube-api-access-6b9x5\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894970-qgxzf\" (UID: \"b1a436ee-2304-4a63-9cd9-82e65871c169\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894970-qgxzf"
Nov 03 09:30:00.403228 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.403092    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blfh8\" (UniqueName: \"kubernetes.io/projected/d0f5712d-329f-4e4e-910f-2351f04f691e-kube-api-access-blfh8\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894970-f5nvf\" (UID: \"d0f5712d-329f-4e4e-910f-2351f04f691e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894970-f5nvf"
Nov 03 09:30:00.403228 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.403111    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s8tl\" (UniqueName: \"kubernetes.io/projected/04abc6b1-3b23-40da-b7ec-5d341ff6ad14-kube-api-access-8s8tl\") pod \"maas-api-key-cleanup-e2e-worker-w2-77207e-29894970-l595n\" (UID: \"04abc6b1-3b23-40da-b7ec-5d341ff6ad14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894970-l595n"
Nov 03 09:30:00.403228 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.403130    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04abc6b1-3b23-40da-b7ec-5d341ff6ad14-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-77207e-29894970-l595n\" (UID: \"04abc6b1-3b23-40da-b7ec-5d341ff6ad14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894970-l595n"
Nov 03 09:30:00.403228 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.403171    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f055424-32fe-49ae-b6a9-4243eaf0c50b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894970-27x62\" (UID: \"5f055424-32fe-49ae-b6a9-4243eaf0c50b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894970-27x62"
Nov 03 09:30:00.403228 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.403193    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7slg4\" (UniqueName: \"kubernetes.io/projected/475138bd-0332-4a16-b413-272ccf954d5a-kube-api-access-7slg4\") pod \"maas-api-key-cleanup-e2e-worker-w0-910d50-29894970-xq4mw\" (UID: \"475138bd-0332-4a16-b413-272ccf954d5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894970-xq4mw"
Nov 03 09:30:00.403847 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.403262    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e5001f-f741-4337-8767-8297949ac190-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-433821-29894970-4zzr8\" (UID: \"68e5001f-f741-4337-8767-8297949ac190\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894970-4zzr8"
Nov 03 09:30:00.403847 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.403278    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccf23ce8-844c-4924-bd1c-e90e1ea762ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-465707-29894970-4g892\" (UID: \"ccf23ce8-844c-4924-bd1c-e90e1ea762ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894970-4g892"
Nov 03 09:30:00.403847 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.403294    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fg8r\" (UniqueName: \"kubernetes.io/projected/ccf23ce8-844c-4924-bd1c-e90e1ea762ab-kube-api-access-2fg8r\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-465707-29894970-4g892\" (UID: \"ccf23ce8-844c-4924-bd1c-e90e1ea762ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894970-4g892"
Nov 03 09:30:00.403847 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.403320    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/475138bd-0332-4a16-b413-272ccf954d5a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-910d50-29894970-xq4mw\" (UID: \"475138bd-0332-4a16-b413-272ccf954d5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894970-xq4mw"
Nov 03 09:30:00.403847 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.403337    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0f5712d-329f-4e4e-910f-2351f04f691e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894970-f5nvf\" (UID: \"d0f5712d-329f-4e4e-910f-2351f04f691e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894970-f5nvf"
Nov 03 09:30:00.403847 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.403362    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwq44\" (UniqueName: \"kubernetes.io/projected/5f055424-32fe-49ae-b6a9-4243eaf0c50b-kube-api-access-bwq44\") pod \"maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894970-27x62\" (UID: \"5f055424-32fe-49ae-b6a9-4243eaf0c50b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894970-27x62"
Nov 03 09:30:00.403847 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.403380    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phbs2\" (UniqueName: \"kubernetes.io/projected/68e5001f-f741-4337-8767-8297949ac190-kube-api-access-phbs2\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-433821-29894970-4zzr8\" (UID: \"68e5001f-f741-4337-8767-8297949ac190\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894970-4zzr8"
Nov 03 09:30:00.403847 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.403422    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6c7f\" (UniqueName: \"kubernetes.io/projected/b18d69e0-049e-4acd-bb6c-59299939e791-kube-api-access-r6c7f\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894970-swn74\" (UID: \"b18d69e0-049e-4acd-bb6c-59299939e791\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894970-swn74"
Nov 03 09:30:00.404601 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.404571    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b18d69e0-049e-4acd-bb6c-59299939e791-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894970-swn74\" (UID: \"b18d69e0-049e-4acd-bb6c-59299939e791\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894970-swn74"
Nov 03 09:30:00.405316 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.405275    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a436ee-2304-4a63-9cd9-82e65871c169-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894970-qgxzf\" (UID: \"b1a436ee-2304-4a63-9cd9-82e65871c169\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894970-qgxzf"
Nov 03 09:30:00.406798 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.406767    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/475138bd-0332-4a16-b413-272ccf954d5a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-910d50-29894970-xq4mw\" (UID: \"475138bd-0332-4a16-b413-272ccf954d5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894970-xq4mw"
Nov 03 09:30:00.406912 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.406842    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e5001f-f741-4337-8767-8297949ac190-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-433821-29894970-4zzr8\" (UID: \"68e5001f-f741-4337-8767-8297949ac190\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894970-4zzr8"
Nov 03 09:30:00.407100 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.407074    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccf23ce8-844c-4924-bd1c-e90e1ea762ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-465707-29894970-4g892\" (UID: \"ccf23ce8-844c-4924-bd1c-e90e1ea762ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894970-4g892"
Nov 03 09:30:00.407190 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.407101    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f055424-32fe-49ae-b6a9-4243eaf0c50b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894970-27x62\" (UID: \"5f055424-32fe-49ae-b6a9-4243eaf0c50b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894970-27x62"
Nov 03 09:30:00.465826 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.465733    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6c7f\" (UniqueName: \"kubernetes.io/projected/b18d69e0-049e-4acd-bb6c-59299939e791-kube-api-access-r6c7f\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894970-swn74\" (UID: \"b18d69e0-049e-4acd-bb6c-59299939e791\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894970-swn74"
Nov 03 09:30:00.465826 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.465744    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phbs2\" (UniqueName: \"kubernetes.io/projected/68e5001f-f741-4337-8767-8297949ac190-kube-api-access-phbs2\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-433821-29894970-4zzr8\" (UID: \"68e5001f-f741-4337-8767-8297949ac190\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894970-4zzr8"
Nov 03 09:30:00.466106 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.465864    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b9x5\" (UniqueName: \"kubernetes.io/projected/b1a436ee-2304-4a63-9cd9-82e65871c169-kube-api-access-6b9x5\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894970-qgxzf\" (UID: \"b1a436ee-2304-4a63-9cd9-82e65871c169\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894970-qgxzf"
Nov 03 09:30:00.466106 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.465884    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fg8r\" (UniqueName: \"kubernetes.io/projected/ccf23ce8-844c-4924-bd1c-e90e1ea762ab-kube-api-access-2fg8r\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-465707-29894970-4g892\" (UID: \"ccf23ce8-844c-4924-bd1c-e90e1ea762ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894970-4g892"
Nov 03 09:30:00.470449 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.470413    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7slg4\" (UniqueName: \"kubernetes.io/projected/475138bd-0332-4a16-b413-272ccf954d5a-kube-api-access-7slg4\") pod \"maas-api-key-cleanup-e2e-worker-w0-910d50-29894970-xq4mw\" (UID: \"475138bd-0332-4a16-b413-272ccf954d5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894970-xq4mw"
Nov 03 09:30:00.470573 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.470543    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwq44\" (UniqueName: \"kubernetes.io/projected/5f055424-32fe-49ae-b6a9-4243eaf0c50b-kube-api-access-bwq44\") pod \"maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894970-27x62\" (UID: \"5f055424-32fe-49ae-b6a9-4243eaf0c50b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894970-27x62"
Nov 03 09:30:00.504443 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.504408    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc8k5\" (UniqueName: \"kubernetes.io/projected/7f59e915-c5e9-4ca4-b67f-fcdec7f83136-kube-api-access-bc8k5\") pod \"maas-api-key-cleanup-29894970-lkt9m\" (UID: \"7f59e915-c5e9-4ca4-b67f-fcdec7f83136\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894970-lkt9m"
Nov 03 09:30:00.504632 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.504449    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f59e915-c5e9-4ca4-b67f-fcdec7f83136-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894970-lkt9m\" (UID: \"7f59e915-c5e9-4ca4-b67f-fcdec7f83136\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894970-lkt9m"
Nov 03 09:30:00.504632 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.504475    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blfh8\" (UniqueName: \"kubernetes.io/projected/d0f5712d-329f-4e4e-910f-2351f04f691e-kube-api-access-blfh8\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894970-f5nvf\" (UID: \"d0f5712d-329f-4e4e-910f-2351f04f691e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894970-f5nvf"
Nov 03 09:30:00.504632 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.504508    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s8tl\" (UniqueName: \"kubernetes.io/projected/04abc6b1-3b23-40da-b7ec-5d341ff6ad14-kube-api-access-8s8tl\") pod \"maas-api-key-cleanup-e2e-worker-w2-77207e-29894970-l595n\" (UID: \"04abc6b1-3b23-40da-b7ec-5d341ff6ad14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894970-l595n"
Nov 03 09:30:00.504632 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.504528    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04abc6b1-3b23-40da-b7ec-5d341ff6ad14-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-77207e-29894970-l595n\" (UID: \"04abc6b1-3b23-40da-b7ec-5d341ff6ad14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894970-l595n"
Nov 03 09:30:00.504856 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.504674    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0f5712d-329f-4e4e-910f-2351f04f691e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894970-f5nvf\" (UID: \"d0f5712d-329f-4e4e-910f-2351f04f691e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894970-f5nvf"
Nov 03 09:30:00.505306 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.505287    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0f5712d-329f-4e4e-910f-2351f04f691e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894970-f5nvf\" (UID: \"d0f5712d-329f-4e4e-910f-2351f04f691e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894970-f5nvf"
Nov 03 09:30:00.505351 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.505297    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f59e915-c5e9-4ca4-b67f-fcdec7f83136-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894970-lkt9m\" (UID: \"7f59e915-c5e9-4ca4-b67f-fcdec7f83136\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894970-lkt9m"
Nov 03 09:30:00.505386 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.505350    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04abc6b1-3b23-40da-b7ec-5d341ff6ad14-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-77207e-29894970-l595n\" (UID: \"04abc6b1-3b23-40da-b7ec-5d341ff6ad14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894970-l595n"
Nov 03 09:30:00.524137 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.524109    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894970-swn74"
Nov 03 09:30:00.537286 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.533703    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894970-27x62"
Nov 03 09:30:00.543693 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.543661    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894970-4zzr8"
Nov 03 09:30:00.575072 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.575020    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894970-xq4mw"
Nov 03 09:30:00.577051 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.576361    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s8tl\" (UniqueName: \"kubernetes.io/projected/04abc6b1-3b23-40da-b7ec-5d341ff6ad14-kube-api-access-8s8tl\") pod \"maas-api-key-cleanup-e2e-worker-w2-77207e-29894970-l595n\" (UID: \"04abc6b1-3b23-40da-b7ec-5d341ff6ad14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894970-l595n"
Nov 03 09:30:00.578451 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.578426    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blfh8\" (UniqueName: \"kubernetes.io/projected/d0f5712d-329f-4e4e-910f-2351f04f691e-kube-api-access-blfh8\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894970-f5nvf\" (UID: \"d0f5712d-329f-4e4e-910f-2351f04f691e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894970-f5nvf"
Nov 03 09:30:00.579766 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.579746    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc8k5\" (UniqueName: \"kubernetes.io/projected/7f59e915-c5e9-4ca4-b67f-fcdec7f83136-kube-api-access-bc8k5\") pod \"maas-api-key-cleanup-29894970-lkt9m\" (UID: \"7f59e915-c5e9-4ca4-b67f-fcdec7f83136\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894970-lkt9m"
Nov 03 09:30:00.583302 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.583081    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894970-4g892"
Nov 03 09:30:00.592473 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.592248    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894970-qgxzf"
Nov 03 09:30:00.600932 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.599141    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894970-lkt9m"
Nov 03 09:30:00.636385 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.636096    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894970-f5nvf"
Nov 03 09:30:00.653848 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.653647    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894970-l595n"
Nov 03 09:30:00.944172 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.944142    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894970-swn74"]
Nov 03 09:30:00.946325 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:30:00.946297    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb18d69e0_049e_4acd_bb6c_59299939e791.slice/crio-a861637d747d6d5b2564735b442054e2782b206e2e78c8cde1c75ab11a21fa20 WatchSource:0}: Error finding container a861637d747d6d5b2564735b442054e2782b206e2e78c8cde1c75ab11a21fa20: Status 404 returned error can't find the container with id a861637d747d6d5b2564735b442054e2782b206e2e78c8cde1c75ab11a21fa20
Nov 03 09:30:00.992539 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.992518    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894970-4zzr8"]
Nov 03 09:30:00.994337 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.994303    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894970-swn74" event={"ID":"b18d69e0-049e-4acd-bb6c-59299939e791","Type":"ContainerStarted","Data":"a861637d747d6d5b2564735b442054e2782b206e2e78c8cde1c75ab11a21fa20"}
Nov 03 09:30:00.996508 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:00.996487    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894970-27x62"]
Nov 03 09:30:00.996572 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:30:00.996500    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68e5001f_f741_4337_8767_8297949ac190.slice/crio-39c4a140707de1d4629822fc93ac5216d0006aa4c9e63e4aece325982833535e WatchSource:0}: Error finding container 39c4a140707de1d4629822fc93ac5216d0006aa4c9e63e4aece325982833535e: Status 404 returned error can't find the container with id 39c4a140707de1d4629822fc93ac5216d0006aa4c9e63e4aece325982833535e
Nov 03 09:30:00.997730 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:30:00.997707    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f055424_32fe_49ae_b6a9_4243eaf0c50b.slice/crio-c456acfe0f25258a54053255c3bb7ccf6133b643708657d0212eaa8769854aae WatchSource:0}: Error finding container c456acfe0f25258a54053255c3bb7ccf6133b643708657d0212eaa8769854aae: Status 404 returned error can't find the container with id c456acfe0f25258a54053255c3bb7ccf6133b643708657d0212eaa8769854aae
Nov 03 09:30:01.049202 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:01.049181    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894970-4g892"]
Nov 03 09:30:01.052574 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:30:01.052548    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccf23ce8_844c_4924_bd1c_e90e1ea762ab.slice/crio-629168844efa47747a29067b27070c3675fe29203dc2092981e9aa7f5ec9d187 WatchSource:0}: Error finding container 629168844efa47747a29067b27070c3675fe29203dc2092981e9aa7f5ec9d187: Status 404 returned error can't find the container with id 629168844efa47747a29067b27070c3675fe29203dc2092981e9aa7f5ec9d187
Nov 03 09:30:01.053081 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:01.053058    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894970-xq4mw"]
Nov 03 09:30:01.080484 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:01.080458    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894970-qgxzf"]
Nov 03 09:30:01.096175 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:30:01.096149    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1a436ee_2304_4a63_9cd9_82e65871c169.slice/crio-fcf9880f7742bc7457510b32bafd2fef0df96527ed7e1227f142f5809ae29369 WatchSource:0}: Error finding container fcf9880f7742bc7457510b32bafd2fef0df96527ed7e1227f142f5809ae29369: Status 404 returned error can't find the container with id fcf9880f7742bc7457510b32bafd2fef0df96527ed7e1227f142f5809ae29369
Nov 03 09:30:01.151770 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:01.151736    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894970-f5nvf"]
Nov 03 09:30:01.156304 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:01.156282    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-465707/auth-test-model-465707-kserve-75c6b79fbf-4v284"]
Nov 03 09:30:01.160896 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:01.160872    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894970-l595n"]
Nov 03 09:30:01.191012 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:01.190976    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894970-lkt9m"]
Nov 03 09:30:01.230491 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:30:01.230460    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04abc6b1_3b23_40da_b7ec_5d341ff6ad14.slice/crio-ed2b03dbb0d5e8efc29fadca18162a3f9660d4bb16b4d70f0821fd6f4001e520 WatchSource:0}: Error finding container ed2b03dbb0d5e8efc29fadca18162a3f9660d4bb16b4d70f0821fd6f4001e520: Status 404 returned error can't find the container with id ed2b03dbb0d5e8efc29fadca18162a3f9660d4bb16b4d70f0821fd6f4001e520
Nov 03 09:30:01.238431 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:30:01.237152    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f59e915_c5e9_4ca4_b67f_fcdec7f83136.slice/crio-4fa1cccfc9532cf56f43ee39ed14813a7220deae1b5b54036cdab54995b98495 WatchSource:0}: Error finding container 4fa1cccfc9532cf56f43ee39ed14813a7220deae1b5b54036cdab54995b98495: Status 404 returned error can't find the container with id 4fa1cccfc9532cf56f43ee39ed14813a7220deae1b5b54036cdab54995b98495
Nov 03 09:30:02.002143 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:02.002106    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894970-4g892" event={"ID":"ccf23ce8-844c-4924-bd1c-e90e1ea762ab","Type":"ContainerStarted","Data":"fb30602a6fcc0af80e6f80496e11bbbcf688a4be525d27962849a44f776afcee"}
Nov 03 09:30:02.002305 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:02.002162    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894970-4g892" event={"ID":"ccf23ce8-844c-4924-bd1c-e90e1ea762ab","Type":"ContainerStarted","Data":"629168844efa47747a29067b27070c3675fe29203dc2092981e9aa7f5ec9d187"}
Nov 03 09:30:02.003648 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:02.003615    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894970-lkt9m" event={"ID":"7f59e915-c5e9-4ca4-b67f-fcdec7f83136","Type":"ContainerStarted","Data":"fc8015b6dae46b5167e49fd4ec9170d69fdc7b9b7bef8fda9f022f354fc193ac"}
Nov 03 09:30:02.003777 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:02.003654    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894970-lkt9m" event={"ID":"7f59e915-c5e9-4ca4-b67f-fcdec7f83136","Type":"ContainerStarted","Data":"4fa1cccfc9532cf56f43ee39ed14813a7220deae1b5b54036cdab54995b98495"}
Nov 03 09:30:02.005294 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:02.005267    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894970-27x62" event={"ID":"5f055424-32fe-49ae-b6a9-4243eaf0c50b","Type":"ContainerStarted","Data":"34d270fd52c467ed3116b1fccb84edf9e7ef0d6b530daa004c01c3b38fafc50f"}
Nov 03 09:30:02.005424 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:02.005319    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894970-27x62" event={"ID":"5f055424-32fe-49ae-b6a9-4243eaf0c50b","Type":"ContainerStarted","Data":"c456acfe0f25258a54053255c3bb7ccf6133b643708657d0212eaa8769854aae"}
Nov 03 09:30:02.006935 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:02.006909    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894970-swn74" event={"ID":"b18d69e0-049e-4acd-bb6c-59299939e791","Type":"ContainerStarted","Data":"8b177f432ae635621649e3900a11068868e95cd8de0d9b93fc385ba91ab9063b"}
Nov 03 09:30:02.008504 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:02.008475    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894970-l595n" event={"ID":"04abc6b1-3b23-40da-b7ec-5d341ff6ad14","Type":"ContainerStarted","Data":"5b499c38ae7aee78dcfe76f67fd419f9c3a5d9054f762007ca8588d0446354cd"}
Nov 03 09:30:02.008613 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:02.008516    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894970-l595n" event={"ID":"04abc6b1-3b23-40da-b7ec-5d341ff6ad14","Type":"ContainerStarted","Data":"ed2b03dbb0d5e8efc29fadca18162a3f9660d4bb16b4d70f0821fd6f4001e520"}
Nov 03 09:30:02.009936 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:02.009916    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894970-4zzr8" event={"ID":"68e5001f-f741-4337-8767-8297949ac190","Type":"ContainerStarted","Data":"8c05d9e7e01e59cf632cfafd460cc2748522dfe5d26a57bc2de23852040f8364"}
Nov 03 09:30:02.010132 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:02.009971    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894970-4zzr8" event={"ID":"68e5001f-f741-4337-8767-8297949ac190","Type":"ContainerStarted","Data":"39c4a140707de1d4629822fc93ac5216d0006aa4c9e63e4aece325982833535e"}
Nov 03 09:30:02.011441 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:02.011420    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894970-f5nvf" event={"ID":"d0f5712d-329f-4e4e-910f-2351f04f691e","Type":"ContainerStarted","Data":"c486866d3324f90fae13ac5181c9abb69d56aebcbe4f81ea15d77839aa77ad31"}
Nov 03 09:30:02.011543 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:02.011460    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894970-f5nvf" event={"ID":"d0f5712d-329f-4e4e-910f-2351f04f691e","Type":"ContainerStarted","Data":"7feda9f6031ccc472e6b5d43b4be068a955afc47b0d76660d29dc25a2bad00eb"}
Nov 03 09:30:02.012880 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:02.012858    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894970-qgxzf" event={"ID":"b1a436ee-2304-4a63-9cd9-82e65871c169","Type":"ContainerStarted","Data":"3525ee65aa24b19d8361f85dead991b999a32586aaab6fd59d81e262b2f31139"}
Nov 03 09:30:02.012992 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:02.012901    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894970-qgxzf" event={"ID":"b1a436ee-2304-4a63-9cd9-82e65871c169","Type":"ContainerStarted","Data":"fcf9880f7742bc7457510b32bafd2fef0df96527ed7e1227f142f5809ae29369"}
Nov 03 09:30:02.014479 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:02.014456    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894970-xq4mw" event={"ID":"475138bd-0332-4a16-b413-272ccf954d5a","Type":"ContainerStarted","Data":"f9447ba1a98f17efa36da4c28b003b564b673a417d90a4b188e538fe9d9c2902"}
Nov 03 09:30:02.014571 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:02.014499    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894970-xq4mw" event={"ID":"475138bd-0332-4a16-b413-272ccf954d5a","Type":"ContainerStarted","Data":"150f38d5acae6ae2a4ef468ee43e3851ba7fba3fb51263bb772e161a2259fc17"}
Nov 03 09:30:02.131040 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:02.128319    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894970-4g892" podStartSLOduration=2.128300404 podStartE2EDuration="2.128300404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:30:02.125397128 +0000 UTC m=+3026.394460563" watchObservedRunningTime="2026-11-03 09:30:02.128300404 +0000 UTC m=+3026.397363840"
Nov 03 09:30:02.249023 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:02.246829    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894970-swn74" podStartSLOduration=2.246806837 podStartE2EDuration="2.246806837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:30:02.243935415 +0000 UTC m=+3026.512998850" watchObservedRunningTime="2026-11-03 09:30:02.246806837 +0000 UTC m=+3026.515870262"
Nov 03 09:30:02.367894 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:02.367837    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894970-f5nvf" podStartSLOduration=2.367821169 podStartE2EDuration="2.367821169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:30:02.364524408 +0000 UTC m=+3026.633587843" watchObservedRunningTime="2026-11-03 09:30:02.367821169 +0000 UTC m=+3026.636884605"
Nov 03 09:30:02.481533 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:02.481483    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894970-27x62" podStartSLOduration=2.481467845 podStartE2EDuration="2.481467845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:30:02.479739792 +0000 UTC m=+3026.748803250" watchObservedRunningTime="2026-11-03 09:30:02.481467845 +0000 UTC m=+3026.750531273"
Nov 03 09:30:02.603126 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:02.603069    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894970-l595n" podStartSLOduration=2.603051526 podStartE2EDuration="2.603051526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:30:02.599393399 +0000 UTC m=+3026.868456834" watchObservedRunningTime="2026-11-03 09:30:02.603051526 +0000 UTC m=+3026.872114960"
Nov 03 09:30:02.836352 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:02.836247    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894970-xq4mw" podStartSLOduration=2.8362289819999997 podStartE2EDuration="2.836228982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:30:02.716975913 +0000 UTC m=+3026.986039359" watchObservedRunningTime="2026-11-03 09:30:02.836228982 +0000 UTC m=+3027.105292422"
Nov 03 09:30:02.839924 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:02.839876    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894970-qgxzf" podStartSLOduration=2.8398633479999997 podStartE2EDuration="2.839863348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:30:02.838270132 +0000 UTC m=+3027.107333547" watchObservedRunningTime="2026-11-03 09:30:02.839863348 +0000 UTC m=+3027.108926786"
Nov 03 09:30:02.960199 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:02.960137    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894970-lkt9m" podStartSLOduration=2.9601168060000003 podStartE2EDuration="2.960116806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:30:02.954385833 +0000 UTC m=+3027.223449292" watchObservedRunningTime="2026-11-03 09:30:02.960116806 +0000 UTC m=+3027.229180242"
Nov 03 09:30:03.029804 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:03.029771    4473 generic.go:356] "Generic (PLEG): container finished" podID="d0f5712d-329f-4e4e-910f-2351f04f691e" containerID="c486866d3324f90fae13ac5181c9abb69d56aebcbe4f81ea15d77839aa77ad31" exitCode=60
Nov 03 09:30:03.030048 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:03.029852    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894970-f5nvf" event={"ID":"d0f5712d-329f-4e4e-910f-2351f04f691e","Type":"ContainerDied","Data":"c486866d3324f90fae13ac5181c9abb69d56aebcbe4f81ea15d77839aa77ad31"}
Nov 03 09:30:03.030168 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:03.030152    4473 scope.go:122] "RemoveContainer" containerID="c486866d3324f90fae13ac5181c9abb69d56aebcbe4f81ea15d77839aa77ad31"
Nov 03 09:30:03.031666 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:03.031623    4473 generic.go:356] "Generic (PLEG): container finished" podID="b1a436ee-2304-4a63-9cd9-82e65871c169" containerID="3525ee65aa24b19d8361f85dead991b999a32586aaab6fd59d81e262b2f31139" exitCode=60
Nov 03 09:30:03.031802 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:03.031726    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894970-qgxzf" event={"ID":"b1a436ee-2304-4a63-9cd9-82e65871c169","Type":"ContainerDied","Data":"3525ee65aa24b19d8361f85dead991b999a32586aaab6fd59d81e262b2f31139"}
Nov 03 09:30:03.032044 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:03.032024    4473 scope.go:122] "RemoveContainer" containerID="3525ee65aa24b19d8361f85dead991b999a32586aaab6fd59d81e262b2f31139"
Nov 03 09:30:03.033869 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:03.033750    4473 generic.go:356] "Generic (PLEG): container finished" podID="475138bd-0332-4a16-b413-272ccf954d5a" containerID="f9447ba1a98f17efa36da4c28b003b564b673a417d90a4b188e538fe9d9c2902" exitCode=60
Nov 03 09:30:03.033869 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:03.033837    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894970-xq4mw" event={"ID":"475138bd-0332-4a16-b413-272ccf954d5a","Type":"ContainerDied","Data":"f9447ba1a98f17efa36da4c28b003b564b673a417d90a4b188e538fe9d9c2902"}
Nov 03 09:30:03.034128 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:03.034111    4473 scope.go:122] "RemoveContainer" containerID="f9447ba1a98f17efa36da4c28b003b564b673a417d90a4b188e538fe9d9c2902"
Nov 03 09:30:03.036666 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:03.036643    4473 generic.go:356] "Generic (PLEG): container finished" podID="ccf23ce8-844c-4924-bd1c-e90e1ea762ab" containerID="fb30602a6fcc0af80e6f80496e11bbbcf688a4be525d27962849a44f776afcee" exitCode=60
Nov 03 09:30:03.036769 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:03.036726    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894970-4g892" event={"ID":"ccf23ce8-844c-4924-bd1c-e90e1ea762ab","Type":"ContainerDied","Data":"fb30602a6fcc0af80e6f80496e11bbbcf688a4be525d27962849a44f776afcee"}
Nov 03 09:30:03.037087 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:03.037066    4473 scope.go:122] "RemoveContainer" containerID="fb30602a6fcc0af80e6f80496e11bbbcf688a4be525d27962849a44f776afcee"
Nov 03 09:30:03.038837 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:03.038817    4473 generic.go:356] "Generic (PLEG): container finished" podID="7f59e915-c5e9-4ca4-b67f-fcdec7f83136" containerID="fc8015b6dae46b5167e49fd4ec9170d69fdc7b9b7bef8fda9f022f354fc193ac" exitCode=60
Nov 03 09:30:03.038968 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:03.038880    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894970-lkt9m" event={"ID":"7f59e915-c5e9-4ca4-b67f-fcdec7f83136","Type":"ContainerDied","Data":"fc8015b6dae46b5167e49fd4ec9170d69fdc7b9b7bef8fda9f022f354fc193ac"}
Nov 03 09:30:03.039146 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:03.039129    4473 scope.go:122] "RemoveContainer" containerID="fc8015b6dae46b5167e49fd4ec9170d69fdc7b9b7bef8fda9f022f354fc193ac"
Nov 03 09:30:03.040712 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:03.040651    4473 generic.go:356] "Generic (PLEG): container finished" podID="5f055424-32fe-49ae-b6a9-4243eaf0c50b" containerID="34d270fd52c467ed3116b1fccb84edf9e7ef0d6b530daa004c01c3b38fafc50f" exitCode=60
Nov 03 09:30:03.040800 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:03.040739    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894970-27x62" event={"ID":"5f055424-32fe-49ae-b6a9-4243eaf0c50b","Type":"ContainerDied","Data":"34d270fd52c467ed3116b1fccb84edf9e7ef0d6b530daa004c01c3b38fafc50f"}
Nov 03 09:30:03.041144 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:03.041126    4473 scope.go:122] "RemoveContainer" containerID="34d270fd52c467ed3116b1fccb84edf9e7ef0d6b530daa004c01c3b38fafc50f"
Nov 03 09:30:03.043056 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:03.043034    4473 generic.go:356] "Generic (PLEG): container finished" podID="b18d69e0-049e-4acd-bb6c-59299939e791" containerID="8b177f432ae635621649e3900a11068868e95cd8de0d9b93fc385ba91ab9063b" exitCode=60
Nov 03 09:30:03.043155 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:03.043101    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894970-swn74" event={"ID":"b18d69e0-049e-4acd-bb6c-59299939e791","Type":"ContainerDied","Data":"8b177f432ae635621649e3900a11068868e95cd8de0d9b93fc385ba91ab9063b"}
Nov 03 09:30:03.043297 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:03.043283    4473 scope.go:122] "RemoveContainer" containerID="8b177f432ae635621649e3900a11068868e95cd8de0d9b93fc385ba91ab9063b"
Nov 03 09:30:03.044750 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:03.044732    4473 generic.go:356] "Generic (PLEG): container finished" podID="04abc6b1-3b23-40da-b7ec-5d341ff6ad14" containerID="5b499c38ae7aee78dcfe76f67fd419f9c3a5d9054f762007ca8588d0446354cd" exitCode=60
Nov 03 09:30:03.044867 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:03.044826    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894970-l595n" event={"ID":"04abc6b1-3b23-40da-b7ec-5d341ff6ad14","Type":"ContainerDied","Data":"5b499c38ae7aee78dcfe76f67fd419f9c3a5d9054f762007ca8588d0446354cd"}
Nov 03 09:30:03.045024 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:03.045007    4473 scope.go:122] "RemoveContainer" containerID="5b499c38ae7aee78dcfe76f67fd419f9c3a5d9054f762007ca8588d0446354cd"
Nov 03 09:30:03.046590 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:03.046567    4473 generic.go:356] "Generic (PLEG): container finished" podID="68e5001f-f741-4337-8767-8297949ac190" containerID="8c05d9e7e01e59cf632cfafd460cc2748522dfe5d26a57bc2de23852040f8364" exitCode=60
Nov 03 09:30:03.046705 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:03.046617    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894970-4zzr8" event={"ID":"68e5001f-f741-4337-8767-8297949ac190","Type":"ContainerDied","Data":"8c05d9e7e01e59cf632cfafd460cc2748522dfe5d26a57bc2de23852040f8364"}
Nov 03 09:30:03.046856 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:03.046838    4473 scope.go:122] "RemoveContainer" containerID="8c05d9e7e01e59cf632cfafd460cc2748522dfe5d26a57bc2de23852040f8364"
Nov 03 09:30:03.074565 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:03.074508    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894970-4zzr8" podStartSLOduration=3.074489511 podStartE2EDuration="3.074489511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:30:03.071885173 +0000 UTC m=+3027.340948608" watchObservedRunningTime="2026-11-03 09:30:03.074489511 +0000 UTC m=+3027.343552947"
Nov 03 09:30:04.055822 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:04.055741    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894970-xq4mw" event={"ID":"475138bd-0332-4a16-b413-272ccf954d5a","Type":"ContainerStarted","Data":"c5e6f6702d0f351b65ee99342a6fac1700514a8be7bcdde72b49801c02852a8c"}
Nov 03 09:30:04.057579 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:04.057557    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894970-4g892" event={"ID":"ccf23ce8-844c-4924-bd1c-e90e1ea762ab","Type":"ContainerStarted","Data":"69ed0fad1f777d7ea9b7c683009246ca6b9bb1ece6f0af52bf32f01cda41e3db"}
Nov 03 09:30:04.059462 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:04.059440    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894970-lkt9m" event={"ID":"7f59e915-c5e9-4ca4-b67f-fcdec7f83136","Type":"ContainerStarted","Data":"8eddf1f0d68391bbece2deb14412068347b7b8df14338ebc73bd3c53b517e2df"}
Nov 03 09:30:04.061312 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:04.061282    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894970-27x62" event={"ID":"5f055424-32fe-49ae-b6a9-4243eaf0c50b","Type":"ContainerStarted","Data":"83f5de433e69f35bd5a24a8e0186bc25d8f444a8f156525ebe039e3043a760dc"}
Nov 03 09:30:04.063244 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:04.063216    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894970-swn74" event={"ID":"b18d69e0-049e-4acd-bb6c-59299939e791","Type":"ContainerStarted","Data":"e02f96b9634425dddf7f0c9656f78f28dcd43b7d6bdee30cedff5737039885d5"}
Nov 03 09:30:04.064994 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:04.064959    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894970-l595n" event={"ID":"04abc6b1-3b23-40da-b7ec-5d341ff6ad14","Type":"ContainerStarted","Data":"91e030b5e320e1144e7e23fd8898b3039e4474abb85f97b7edaad0ced1fb17cb"}
Nov 03 09:30:04.066845 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:04.066820    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894970-4zzr8" event={"ID":"68e5001f-f741-4337-8767-8297949ac190","Type":"ContainerStarted","Data":"e4a8270e31f3a349cc7d95f6b313c029c2d9869906dd0da605dadd9f6dd311a6"}
Nov 03 09:30:04.068686 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:04.068662    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894970-f5nvf" event={"ID":"d0f5712d-329f-4e4e-910f-2351f04f691e","Type":"ContainerStarted","Data":"08c62caa2b497c6dac2c7200e0d7bcc23f2e6b908a9c5cc3921e69ab4582998b"}
Nov 03 09:30:04.070517 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:04.070496    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894970-qgxzf" event={"ID":"b1a436ee-2304-4a63-9cd9-82e65871c169","Type":"ContainerStarted","Data":"aaea2e9b71699bf2b0e00222719b09a0e7f89b055579ed6723895ed55d12f698"}
Nov 03 09:30:05.080211 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:05.080177    4473 generic.go:356] "Generic (PLEG): container finished" podID="04abc6b1-3b23-40da-b7ec-5d341ff6ad14" containerID="91e030b5e320e1144e7e23fd8898b3039e4474abb85f97b7edaad0ced1fb17cb" exitCode=60
Nov 03 09:30:05.080620 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:05.080253    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894970-l595n" event={"ID":"04abc6b1-3b23-40da-b7ec-5d341ff6ad14","Type":"ContainerDied","Data":"91e030b5e320e1144e7e23fd8898b3039e4474abb85f97b7edaad0ced1fb17cb"}
Nov 03 09:30:05.080620 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:05.080314    4473 scope.go:122] "RemoveContainer" containerID="5b499c38ae7aee78dcfe76f67fd419f9c3a5d9054f762007ca8588d0446354cd"
Nov 03 09:30:05.080620 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:05.080573    4473 scope.go:122] "RemoveContainer" containerID="91e030b5e320e1144e7e23fd8898b3039e4474abb85f97b7edaad0ced1fb17cb"
Nov 03 09:30:05.080871 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:30:05.080849    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-77207e-29894970-l595n_odh-ai-gateway-infra(04abc6b1-3b23-40da-b7ec-5d341ff6ad14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894970-l595n" podUID="04abc6b1-3b23-40da-b7ec-5d341ff6ad14"
Nov 03 09:30:05.082381 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:05.082362    4473 generic.go:356] "Generic (PLEG): container finished" podID="68e5001f-f741-4337-8767-8297949ac190" containerID="e4a8270e31f3a349cc7d95f6b313c029c2d9869906dd0da605dadd9f6dd311a6" exitCode=60
Nov 03 09:30:05.082499 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:05.082442    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894970-4zzr8" event={"ID":"68e5001f-f741-4337-8767-8297949ac190","Type":"ContainerDied","Data":"e4a8270e31f3a349cc7d95f6b313c029c2d9869906dd0da605dadd9f6dd311a6"}
Nov 03 09:30:05.082648 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:05.082635    4473 scope.go:122] "RemoveContainer" containerID="e4a8270e31f3a349cc7d95f6b313c029c2d9869906dd0da605dadd9f6dd311a6"
Nov 03 09:30:05.082882 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:30:05.082863    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-433821-29894970-4zzr8_odh-ai-gateway-infra(68e5001f-f741-4337-8767-8297949ac190)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894970-4zzr8" podUID="68e5001f-f741-4337-8767-8297949ac190"
Nov 03 09:30:05.084346 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:05.084325    4473 generic.go:356] "Generic (PLEG): container finished" podID="d0f5712d-329f-4e4e-910f-2351f04f691e" containerID="08c62caa2b497c6dac2c7200e0d7bcc23f2e6b908a9c5cc3921e69ab4582998b" exitCode=60
Nov 03 09:30:05.084456 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:05.084396    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894970-f5nvf" event={"ID":"d0f5712d-329f-4e4e-910f-2351f04f691e","Type":"ContainerDied","Data":"08c62caa2b497c6dac2c7200e0d7bcc23f2e6b908a9c5cc3921e69ab4582998b"}
Nov 03 09:30:05.084644 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:05.084628    4473 scope.go:122] "RemoveContainer" containerID="08c62caa2b497c6dac2c7200e0d7bcc23f2e6b908a9c5cc3921e69ab4582998b"
Nov 03 09:30:05.084868 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:30:05.084847    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894970-f5nvf_odh-ai-gateway-infra(d0f5712d-329f-4e4e-910f-2351f04f691e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894970-f5nvf" podUID="d0f5712d-329f-4e4e-910f-2351f04f691e"
Nov 03 09:30:05.086491 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:05.086472    4473 generic.go:356] "Generic (PLEG): container finished" podID="b1a436ee-2304-4a63-9cd9-82e65871c169" containerID="aaea2e9b71699bf2b0e00222719b09a0e7f89b055579ed6723895ed55d12f698" exitCode=60
Nov 03 09:30:05.086578 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:05.086553    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894970-qgxzf" event={"ID":"b1a436ee-2304-4a63-9cd9-82e65871c169","Type":"ContainerDied","Data":"aaea2e9b71699bf2b0e00222719b09a0e7f89b055579ed6723895ed55d12f698"}
Nov 03 09:30:05.086801 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:05.086787    4473 scope.go:122] "RemoveContainer" containerID="aaea2e9b71699bf2b0e00222719b09a0e7f89b055579ed6723895ed55d12f698"
Nov 03 09:30:05.087040 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:30:05.087013    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894970-qgxzf_odh-ai-gateway-infra(b1a436ee-2304-4a63-9cd9-82e65871c169)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894970-qgxzf" podUID="b1a436ee-2304-4a63-9cd9-82e65871c169"
Nov 03 09:30:05.088538 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:05.088517    4473 generic.go:356] "Generic (PLEG): container finished" podID="475138bd-0332-4a16-b413-272ccf954d5a" containerID="c5e6f6702d0f351b65ee99342a6fac1700514a8be7bcdde72b49801c02852a8c" exitCode=60
Nov 03 09:30:05.088647 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:05.088597    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894970-xq4mw" event={"ID":"475138bd-0332-4a16-b413-272ccf954d5a","Type":"ContainerDied","Data":"c5e6f6702d0f351b65ee99342a6fac1700514a8be7bcdde72b49801c02852a8c"}
Nov 03 09:30:05.088806 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:05.088789    4473 scope.go:122] "RemoveContainer" containerID="c5e6f6702d0f351b65ee99342a6fac1700514a8be7bcdde72b49801c02852a8c"
Nov 03 09:30:05.089039 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:30:05.089018    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-910d50-29894970-xq4mw_odh-ai-gateway-infra(475138bd-0332-4a16-b413-272ccf954d5a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894970-xq4mw" podUID="475138bd-0332-4a16-b413-272ccf954d5a"
Nov 03 09:30:05.090506 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:05.090488    4473 generic.go:356] "Generic (PLEG): container finished" podID="ccf23ce8-844c-4924-bd1c-e90e1ea762ab" containerID="69ed0fad1f777d7ea9b7c683009246ca6b9bb1ece6f0af52bf32f01cda41e3db" exitCode=60
Nov 03 09:30:05.090609 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:05.090531    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894970-4g892" event={"ID":"ccf23ce8-844c-4924-bd1c-e90e1ea762ab","Type":"ContainerDied","Data":"69ed0fad1f777d7ea9b7c683009246ca6b9bb1ece6f0af52bf32f01cda41e3db"}
Nov 03 09:30:05.090797 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:05.090780    4473 scope.go:122] "RemoveContainer" containerID="69ed0fad1f777d7ea9b7c683009246ca6b9bb1ece6f0af52bf32f01cda41e3db"
Nov 03 09:30:05.091066 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:30:05.091040    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-465707-29894970-4g892_odh-ai-gateway-infra(ccf23ce8-844c-4924-bd1c-e90e1ea762ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894970-4g892" podUID="ccf23ce8-844c-4924-bd1c-e90e1ea762ab"
Nov 03 09:30:05.092500 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:05.092480    4473 generic.go:356] "Generic (PLEG): container finished" podID="7f59e915-c5e9-4ca4-b67f-fcdec7f83136" containerID="8eddf1f0d68391bbece2deb14412068347b7b8df14338ebc73bd3c53b517e2df" exitCode=60
Nov 03 09:30:05.092582 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:05.092560    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894970-lkt9m" event={"ID":"7f59e915-c5e9-4ca4-b67f-fcdec7f83136","Type":"ContainerDied","Data":"8eddf1f0d68391bbece2deb14412068347b7b8df14338ebc73bd3c53b517e2df"}
Nov 03 09:30:05.092748 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:05.092730    4473 scope.go:122] "RemoveContainer" containerID="8eddf1f0d68391bbece2deb14412068347b7b8df14338ebc73bd3c53b517e2df"
Nov 03 09:30:05.092979 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:30:05.092932    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894970-lkt9m_odh-ai-gateway-infra(7f59e915-c5e9-4ca4-b67f-fcdec7f83136)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894970-lkt9m" podUID="7f59e915-c5e9-4ca4-b67f-fcdec7f83136"
Nov 03 09:30:05.094389 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:05.094371    4473 generic.go:356] "Generic (PLEG): container finished" podID="5f055424-32fe-49ae-b6a9-4243eaf0c50b" containerID="83f5de433e69f35bd5a24a8e0186bc25d8f444a8f156525ebe039e3043a760dc" exitCode=60
Nov 03 09:30:05.094475 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:05.094458    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894970-27x62" event={"ID":"5f055424-32fe-49ae-b6a9-4243eaf0c50b","Type":"ContainerDied","Data":"83f5de433e69f35bd5a24a8e0186bc25d8f444a8f156525ebe039e3043a760dc"}
Nov 03 09:30:05.094606 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:05.094576    4473 scope.go:122] "RemoveContainer" containerID="8c05d9e7e01e59cf632cfafd460cc2748522dfe5d26a57bc2de23852040f8364"
Nov 03 09:30:05.094606 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:05.094601    4473 scope.go:122] "RemoveContainer" containerID="83f5de433e69f35bd5a24a8e0186bc25d8f444a8f156525ebe039e3043a760dc"
Nov 03 09:30:05.094816 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:30:05.094790    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894970-27x62_odh-ai-gateway-infra(5f055424-32fe-49ae-b6a9-4243eaf0c50b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894970-27x62" podUID="5f055424-32fe-49ae-b6a9-4243eaf0c50b"
Nov 03 09:30:05.096324 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:05.096303    4473 generic.go:356] "Generic (PLEG): container finished" podID="b18d69e0-049e-4acd-bb6c-59299939e791" containerID="e02f96b9634425dddf7f0c9656f78f28dcd43b7d6bdee30cedff5737039885d5" exitCode=60
Nov 03 09:30:05.096414 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:05.096338    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894970-swn74" event={"ID":"b18d69e0-049e-4acd-bb6c-59299939e791","Type":"ContainerDied","Data":"e02f96b9634425dddf7f0c9656f78f28dcd43b7d6bdee30cedff5737039885d5"}
Nov 03 09:30:05.096528 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:05.096516    4473 scope.go:122] "RemoveContainer" containerID="e02f96b9634425dddf7f0c9656f78f28dcd43b7d6bdee30cedff5737039885d5"
Nov 03 09:30:05.096691 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:30:05.096674    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894970-swn74_odh-ai-gateway-infra(b18d69e0-049e-4acd-bb6c-59299939e791)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894970-swn74" podUID="b18d69e0-049e-4acd-bb6c-59299939e791"
Nov 03 09:30:05.107896 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:05.107876    4473 scope.go:122] "RemoveContainer" containerID="c486866d3324f90fae13ac5181c9abb69d56aebcbe4f81ea15d77839aa77ad31"
Nov 03 09:30:05.120238 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:05.120215    4473 scope.go:122] "RemoveContainer" containerID="3525ee65aa24b19d8361f85dead991b999a32586aaab6fd59d81e262b2f31139"
Nov 03 09:30:05.132436 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:05.132420    4473 scope.go:122] "RemoveContainer" containerID="f9447ba1a98f17efa36da4c28b003b564b673a417d90a4b188e538fe9d9c2902"
Nov 03 09:30:05.144457 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:05.144438    4473 scope.go:122] "RemoveContainer" containerID="fb30602a6fcc0af80e6f80496e11bbbcf688a4be525d27962849a44f776afcee"
Nov 03 09:30:05.156879 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:05.156853    4473 scope.go:122] "RemoveContainer" containerID="fc8015b6dae46b5167e49fd4ec9170d69fdc7b9b7bef8fda9f022f354fc193ac"
Nov 03 09:30:05.170380 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:05.170361    4473 scope.go:122] "RemoveContainer" containerID="34d270fd52c467ed3116b1fccb84edf9e7ef0d6b530daa004c01c3b38fafc50f"
Nov 03 09:30:05.184210 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:05.184194    4473 scope.go:122] "RemoveContainer" containerID="8b177f432ae635621649e3900a11068868e95cd8de0d9b93fc385ba91ab9063b"
Nov 03 09:30:06.104778 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:06.104752    4473 scope.go:122] "RemoveContainer" containerID="08c62caa2b497c6dac2c7200e0d7bcc23f2e6b908a9c5cc3921e69ab4582998b"
Nov 03 09:30:06.105247 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:30:06.105059    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894970-f5nvf_odh-ai-gateway-infra(d0f5712d-329f-4e4e-910f-2351f04f691e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894970-f5nvf" podUID="d0f5712d-329f-4e4e-910f-2351f04f691e"
Nov 03 09:30:06.106492 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:06.106472    4473 scope.go:122] "RemoveContainer" containerID="aaea2e9b71699bf2b0e00222719b09a0e7f89b055579ed6723895ed55d12f698"
Nov 03 09:30:06.106744 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:30:06.106723    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894970-qgxzf_odh-ai-gateway-infra(b1a436ee-2304-4a63-9cd9-82e65871c169)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894970-qgxzf" podUID="b1a436ee-2304-4a63-9cd9-82e65871c169"
Nov 03 09:30:06.108681 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:06.108658    4473 scope.go:122] "RemoveContainer" containerID="c5e6f6702d0f351b65ee99342a6fac1700514a8be7bcdde72b49801c02852a8c"
Nov 03 09:30:06.108962 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:30:06.108922    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-910d50-29894970-xq4mw_odh-ai-gateway-infra(475138bd-0332-4a16-b413-272ccf954d5a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894970-xq4mw" podUID="475138bd-0332-4a16-b413-272ccf954d5a"
Nov 03 09:30:06.110289 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:06.110271    4473 scope.go:122] "RemoveContainer" containerID="69ed0fad1f777d7ea9b7c683009246ca6b9bb1ece6f0af52bf32f01cda41e3db"
Nov 03 09:30:06.110517 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:30:06.110495    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-465707-29894970-4g892_odh-ai-gateway-infra(ccf23ce8-844c-4924-bd1c-e90e1ea762ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894970-4g892" podUID="ccf23ce8-844c-4924-bd1c-e90e1ea762ab"
Nov 03 09:30:06.111929 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:06.111913    4473 scope.go:122] "RemoveContainer" containerID="8eddf1f0d68391bbece2deb14412068347b7b8df14338ebc73bd3c53b517e2df"
Nov 03 09:30:06.112136 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:30:06.112113    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894970-lkt9m_odh-ai-gateway-infra(7f59e915-c5e9-4ca4-b67f-fcdec7f83136)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894970-lkt9m" podUID="7f59e915-c5e9-4ca4-b67f-fcdec7f83136"
Nov 03 09:30:06.113423 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:06.113405    4473 scope.go:122] "RemoveContainer" containerID="83f5de433e69f35bd5a24a8e0186bc25d8f444a8f156525ebe039e3043a760dc"
Nov 03 09:30:06.113596 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:30:06.113575    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894970-27x62_odh-ai-gateway-infra(5f055424-32fe-49ae-b6a9-4243eaf0c50b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894970-27x62" podUID="5f055424-32fe-49ae-b6a9-4243eaf0c50b"
Nov 03 09:30:06.114911 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:06.114892    4473 scope.go:122] "RemoveContainer" containerID="e02f96b9634425dddf7f0c9656f78f28dcd43b7d6bdee30cedff5737039885d5"
Nov 03 09:30:06.115120 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:30:06.115097    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894970-swn74_odh-ai-gateway-infra(b18d69e0-049e-4acd-bb6c-59299939e791)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894970-swn74" podUID="b18d69e0-049e-4acd-bb6c-59299939e791"
Nov 03 09:30:06.116384 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:06.116366    4473 scope.go:122] "RemoveContainer" containerID="91e030b5e320e1144e7e23fd8898b3039e4474abb85f97b7edaad0ced1fb17cb"
Nov 03 09:30:06.116552 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:30:06.116535    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-77207e-29894970-l595n_odh-ai-gateway-infra(04abc6b1-3b23-40da-b7ec-5d341ff6ad14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894970-l595n" podUID="04abc6b1-3b23-40da-b7ec-5d341ff6ad14"
Nov 03 09:30:06.118226 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:06.118212    4473 scope.go:122] "RemoveContainer" containerID="e4a8270e31f3a349cc7d95f6b313c029c2d9869906dd0da605dadd9f6dd311a6"
Nov 03 09:30:06.118395 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:30:06.118378    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-433821-29894970-4zzr8_odh-ai-gateway-infra(68e5001f-f741-4337-8767-8297949ac190)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894970-4zzr8" podUID="68e5001f-f741-4337-8767-8297949ac190"
Nov 03 09:30:13.899821 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:13.899785    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-433821/auth-test-model-433821-kserve-b9dc66df-7psmx" podUID="e99dc6a6-6192-473b-a7b2-27cc028c773c" containerName="main" containerID="cri-o://80bec7f1d508d86ac9da68c68e4ef4c8b53f74acc335e590da80ae59ebff4b22" gracePeriod=45
Nov 03 09:30:14.166746 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:14.166684    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-433821/auth-test-model-433821-kserve-b9dc66df-7psmx"
Nov 03 09:30:14.174572 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:14.174544    4473 generic.go:356] "Generic (PLEG): container finished" podID="e99dc6a6-6192-473b-a7b2-27cc028c773c" containerID="80bec7f1d508d86ac9da68c68e4ef4c8b53f74acc335e590da80ae59ebff4b22" exitCode=0
Nov 03 09:30:14.174704 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:14.174580    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-433821/auth-test-model-433821-kserve-b9dc66df-7psmx" event={"ID":"e99dc6a6-6192-473b-a7b2-27cc028c773c","Type":"ContainerDied","Data":"80bec7f1d508d86ac9da68c68e4ef4c8b53f74acc335e590da80ae59ebff4b22"}
Nov 03 09:30:14.174704 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:14.174591    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-433821/auth-test-model-433821-kserve-b9dc66df-7psmx"
Nov 03 09:30:14.174704 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:14.174621    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-433821/auth-test-model-433821-kserve-b9dc66df-7psmx" event={"ID":"e99dc6a6-6192-473b-a7b2-27cc028c773c","Type":"ContainerDied","Data":"9f073416ee01496841374079b45e7c75a1a69270c31fdaf47077e11951785f12"}
Nov 03 09:30:14.174704 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:14.174639    4473 scope.go:122] "RemoveContainer" containerID="80bec7f1d508d86ac9da68c68e4ef4c8b53f74acc335e590da80ae59ebff4b22"
Nov 03 09:30:14.184177 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:14.184157    4473 scope.go:122] "RemoveContainer" containerID="80bec7f1d508d86ac9da68c68e4ef4c8b53f74acc335e590da80ae59ebff4b22"
Nov 03 09:30:14.184466 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:30:14.184449    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80bec7f1d508d86ac9da68c68e4ef4c8b53f74acc335e590da80ae59ebff4b22\": container with ID starting with 80bec7f1d508d86ac9da68c68e4ef4c8b53f74acc335e590da80ae59ebff4b22 not found: ID does not exist" containerID="80bec7f1d508d86ac9da68c68e4ef4c8b53f74acc335e590da80ae59ebff4b22"
Nov 03 09:30:14.184527 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:14.184475    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80bec7f1d508d86ac9da68c68e4ef4c8b53f74acc335e590da80ae59ebff4b22"} err="failed to get container status \"80bec7f1d508d86ac9da68c68e4ef4c8b53f74acc335e590da80ae59ebff4b22\": rpc error: code = NotFound desc = could not find container \"80bec7f1d508d86ac9da68c68e4ef4c8b53f74acc335e590da80ae59ebff4b22\": container with ID starting with 80bec7f1d508d86ac9da68c68e4ef4c8b53f74acc335e590da80ae59ebff4b22 not found: ID does not exist"
Nov 03 09:30:14.295498 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:14.295472    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e99dc6a6-6192-473b-a7b2-27cc028c773c-kube-api-access-q6xwk\" (UniqueName: \"kubernetes.io/projected/e99dc6a6-6192-473b-a7b2-27cc028c773c-kube-api-access-q6xwk\") pod \"e99dc6a6-6192-473b-a7b2-27cc028c773c\" (UID: \"e99dc6a6-6192-473b-a7b2-27cc028c773c\") "
Nov 03 09:30:14.295680 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:14.295523    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e99dc6a6-6192-473b-a7b2-27cc028c773c-tls-certs\" (UniqueName: \"kubernetes.io/secret/e99dc6a6-6192-473b-a7b2-27cc028c773c-tls-certs\") pod \"e99dc6a6-6192-473b-a7b2-27cc028c773c\" (UID: \"e99dc6a6-6192-473b-a7b2-27cc028c773c\") "
Nov 03 09:30:14.295680 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:14.295626    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e99dc6a6-6192-473b-a7b2-27cc028c773c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e99dc6a6-6192-473b-a7b2-27cc028c773c-tmp-dir\") pod \"e99dc6a6-6192-473b-a7b2-27cc028c773c\" (UID: \"e99dc6a6-6192-473b-a7b2-27cc028c773c\") "
Nov 03 09:30:14.295680 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:14.295658    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e99dc6a6-6192-473b-a7b2-27cc028c773c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e99dc6a6-6192-473b-a7b2-27cc028c773c-model-cache\") pod \"e99dc6a6-6192-473b-a7b2-27cc028c773c\" (UID: \"e99dc6a6-6192-473b-a7b2-27cc028c773c\") "
Nov 03 09:30:14.295860 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:14.295717    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e99dc6a6-6192-473b-a7b2-27cc028c773c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e99dc6a6-6192-473b-a7b2-27cc028c773c-dshm\") pod \"e99dc6a6-6192-473b-a7b2-27cc028c773c\" (UID: \"e99dc6a6-6192-473b-a7b2-27cc028c773c\") "
Nov 03 09:30:14.295860 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:14.295771    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e99dc6a6-6192-473b-a7b2-27cc028c773c-home\" (UniqueName: \"kubernetes.io/empty-dir/e99dc6a6-6192-473b-a7b2-27cc028c773c-home\") pod \"e99dc6a6-6192-473b-a7b2-27cc028c773c\" (UID: \"e99dc6a6-6192-473b-a7b2-27cc028c773c\") "
Nov 03 09:30:14.295988 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:14.295934    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e99dc6a6-6192-473b-a7b2-27cc028c773c-tmp-dir" pod "e99dc6a6-6192-473b-a7b2-27cc028c773c" (UID: "e99dc6a6-6192-473b-a7b2-27cc028c773c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:30:14.296173 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:14.296150    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e99dc6a6-6192-473b-a7b2-27cc028c773c-home" pod "e99dc6a6-6192-473b-a7b2-27cc028c773c" (UID: "e99dc6a6-6192-473b-a7b2-27cc028c773c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:30:14.296258 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:14.296234    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e99dc6a6-6192-473b-a7b2-27cc028c773c-model-cache" pod "e99dc6a6-6192-473b-a7b2-27cc028c773c" (UID: "e99dc6a6-6192-473b-a7b2-27cc028c773c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:30:14.296368 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:14.296353    4473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e99dc6a6-6192-473b-a7b2-27cc028c773c-tmp-dir\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:30:14.296464 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:14.296374    4473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e99dc6a6-6192-473b-a7b2-27cc028c773c-model-cache\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:30:14.296464 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:14.296389    4473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e99dc6a6-6192-473b-a7b2-27cc028c773c-home\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:30:14.298453 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:14.298426    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e99dc6a6-6192-473b-a7b2-27cc028c773c-kube-api-access-q6xwk" pod "e99dc6a6-6192-473b-a7b2-27cc028c773c" (UID: "e99dc6a6-6192-473b-a7b2-27cc028c773c"). InnerVolumeSpecName "kube-api-access-q6xwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:30:14.298549 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:14.298501    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e99dc6a6-6192-473b-a7b2-27cc028c773c-dshm" pod "e99dc6a6-6192-473b-a7b2-27cc028c773c" (UID: "e99dc6a6-6192-473b-a7b2-27cc028c773c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:30:14.298592 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:14.298546    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e99dc6a6-6192-473b-a7b2-27cc028c773c-tls-certs" pod "e99dc6a6-6192-473b-a7b2-27cc028c773c" (UID: "e99dc6a6-6192-473b-a7b2-27cc028c773c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:30:14.397923 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:14.397885    4473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e99dc6a6-6192-473b-a7b2-27cc028c773c-dshm\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:30:14.397923 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:14.397921    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6xwk\" (UniqueName: \"kubernetes.io/projected/e99dc6a6-6192-473b-a7b2-27cc028c773c-kube-api-access-q6xwk\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:30:14.398124 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:14.397937    4473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e99dc6a6-6192-473b-a7b2-27cc028c773c-tls-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:30:14.601307 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:14.601272    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-433821/auth-test-model-433821-kserve-b9dc66df-7psmx"]
Nov 03 09:30:14.606714 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:14.606644    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-433821/auth-test-model-433821-kserve-b9dc66df-7psmx"]
Nov 03 09:30:14.863365 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:14.863289    4473 scope.go:122] "RemoveContainer" containerID="c5e6f6702d0f351b65ee99342a6fac1700514a8be7bcdde72b49801c02852a8c"
Nov 03 09:30:15.184616 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:15.184585    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894970-xq4mw" event={"ID":"475138bd-0332-4a16-b413-272ccf954d5a","Type":"ContainerStarted","Data":"58ff17c95ff6f02e5522f4d0f3977f6c498eea8a1b5917eb97c32dccc8436b61"}
Nov 03 09:30:15.867148 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:15.867107    4473 scope.go:122] "RemoveContainer" containerID="91e030b5e320e1144e7e23fd8898b3039e4474abb85f97b7edaad0ced1fb17cb"
Nov 03 09:30:15.872077 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:15.872057    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e99dc6a6-6192-473b-a7b2-27cc028c773c" path="/var/lib/kubelet/pods/e99dc6a6-6192-473b-a7b2-27cc028c773c/volumes"
Nov 03 09:30:16.199791 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:16.199648    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894970-l595n" event={"ID":"04abc6b1-3b23-40da-b7ec-5d341ff6ad14","Type":"ContainerStarted","Data":"bd192d3a2257535daadeea529c1c35667094e61d102fd3f6c5c5d8e918975800"}
Nov 03 09:30:16.203334 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:16.203306    4473 generic.go:356] "Generic (PLEG): container finished" podID="475138bd-0332-4a16-b413-272ccf954d5a" containerID="58ff17c95ff6f02e5522f4d0f3977f6c498eea8a1b5917eb97c32dccc8436b61" exitCode=60
Nov 03 09:30:16.203593 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:16.203385    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894970-xq4mw" event={"ID":"475138bd-0332-4a16-b413-272ccf954d5a","Type":"ContainerDied","Data":"58ff17c95ff6f02e5522f4d0f3977f6c498eea8a1b5917eb97c32dccc8436b61"}
Nov 03 09:30:16.203593 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:16.203471    4473 scope.go:122] "RemoveContainer" containerID="c5e6f6702d0f351b65ee99342a6fac1700514a8be7bcdde72b49801c02852a8c"
Nov 03 09:30:16.203902 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:16.203887    4473 scope.go:122] "RemoveContainer" containerID="58ff17c95ff6f02e5522f4d0f3977f6c498eea8a1b5917eb97c32dccc8436b61"
Nov 03 09:30:16.204248 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:30:16.204223    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-910d50-29894970-xq4mw_odh-ai-gateway-infra(475138bd-0332-4a16-b413-272ccf954d5a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894970-xq4mw" podUID="475138bd-0332-4a16-b413-272ccf954d5a"
Nov 03 09:30:16.256341 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:16.256310    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-465707/auth-test-model-465707-kserve-75c6b79fbf-4v284" podUID="9cce93be-a3e6-45db-9a17-3560f3822c2e" containerName="main" containerID="cri-o://fc6cacad5619cc79e394867b8fe8a50378b20829082b6b3cbe08882e2c7c4e75" gracePeriod=45
Nov 03 09:30:16.314061 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:16.314034    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894970-xq4mw"]
Nov 03 09:30:16.541219 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:16.541193    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-465707/auth-test-model-465707-kserve-75c6b79fbf-4v284"
Nov 03 09:30:16.624717 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:16.624684    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cce93be-a3e6-45db-9a17-3560f3822c2e-tls-certs\" (UniqueName: \"kubernetes.io/secret/9cce93be-a3e6-45db-9a17-3560f3822c2e-tls-certs\") pod \"9cce93be-a3e6-45db-9a17-3560f3822c2e\" (UID: \"9cce93be-a3e6-45db-9a17-3560f3822c2e\") "
Nov 03 09:30:16.624875 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:16.624752    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cce93be-a3e6-45db-9a17-3560f3822c2e-kube-api-access-rln2b\" (UniqueName: \"kubernetes.io/projected/9cce93be-a3e6-45db-9a17-3560f3822c2e-kube-api-access-rln2b\") pod \"9cce93be-a3e6-45db-9a17-3560f3822c2e\" (UID: \"9cce93be-a3e6-45db-9a17-3560f3822c2e\") "
Nov 03 09:30:16.624920 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:16.624873    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cce93be-a3e6-45db-9a17-3560f3822c2e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cce93be-a3e6-45db-9a17-3560f3822c2e-tmp-dir\") pod \"9cce93be-a3e6-45db-9a17-3560f3822c2e\" (UID: \"9cce93be-a3e6-45db-9a17-3560f3822c2e\") "
Nov 03 09:30:16.625005 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:16.624988    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cce93be-a3e6-45db-9a17-3560f3822c2e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cce93be-a3e6-45db-9a17-3560f3822c2e-dshm\") pod \"9cce93be-a3e6-45db-9a17-3560f3822c2e\" (UID: \"9cce93be-a3e6-45db-9a17-3560f3822c2e\") "
Nov 03 09:30:16.625076 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:16.625018    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cce93be-a3e6-45db-9a17-3560f3822c2e-home\" (UniqueName: \"kubernetes.io/empty-dir/9cce93be-a3e6-45db-9a17-3560f3822c2e-home\") pod \"9cce93be-a3e6-45db-9a17-3560f3822c2e\" (UID: \"9cce93be-a3e6-45db-9a17-3560f3822c2e\") "
Nov 03 09:30:16.625076 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:16.625053    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cce93be-a3e6-45db-9a17-3560f3822c2e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cce93be-a3e6-45db-9a17-3560f3822c2e-model-cache\") pod \"9cce93be-a3e6-45db-9a17-3560f3822c2e\" (UID: \"9cce93be-a3e6-45db-9a17-3560f3822c2e\") "
Nov 03 09:30:16.625183 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:16.625166    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cce93be-a3e6-45db-9a17-3560f3822c2e-tmp-dir" pod "9cce93be-a3e6-45db-9a17-3560f3822c2e" (UID: "9cce93be-a3e6-45db-9a17-3560f3822c2e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:30:16.625308 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:16.625283    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cce93be-a3e6-45db-9a17-3560f3822c2e-home" pod "9cce93be-a3e6-45db-9a17-3560f3822c2e" (UID: "9cce93be-a3e6-45db-9a17-3560f3822c2e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:30:16.625507 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:16.625435    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cce93be-a3e6-45db-9a17-3560f3822c2e-model-cache" pod "9cce93be-a3e6-45db-9a17-3560f3822c2e" (UID: "9cce93be-a3e6-45db-9a17-3560f3822c2e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:30:16.625811 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:16.625790    4473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cce93be-a3e6-45db-9a17-3560f3822c2e-home\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:30:16.625888 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:16.625817    4473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cce93be-a3e6-45db-9a17-3560f3822c2e-model-cache\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:30:16.625888 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:16.625833    4473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cce93be-a3e6-45db-9a17-3560f3822c2e-tmp-dir\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:30:16.627132 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:16.627111    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cce93be-a3e6-45db-9a17-3560f3822c2e-kube-api-access-rln2b" pod "9cce93be-a3e6-45db-9a17-3560f3822c2e" (UID: "9cce93be-a3e6-45db-9a17-3560f3822c2e"). InnerVolumeSpecName "kube-api-access-rln2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:30:16.627132 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:16.627120    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cce93be-a3e6-45db-9a17-3560f3822c2e-dshm" pod "9cce93be-a3e6-45db-9a17-3560f3822c2e" (UID: "9cce93be-a3e6-45db-9a17-3560f3822c2e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:30:16.627272 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:16.627116    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cce93be-a3e6-45db-9a17-3560f3822c2e-tls-certs" pod "9cce93be-a3e6-45db-9a17-3560f3822c2e" (UID: "9cce93be-a3e6-45db-9a17-3560f3822c2e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:30:16.727406 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:16.727378    4473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cce93be-a3e6-45db-9a17-3560f3822c2e-dshm\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:30:16.727515 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:16.727411    4473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cce93be-a3e6-45db-9a17-3560f3822c2e-tls-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:30:16.727515 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:16.727425    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rln2b\" (UniqueName: \"kubernetes.io/projected/9cce93be-a3e6-45db-9a17-3560f3822c2e-kube-api-access-rln2b\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:30:16.862914 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:16.862835    4473 scope.go:122] "RemoveContainer" containerID="08c62caa2b497c6dac2c7200e0d7bcc23f2e6b908a9c5cc3921e69ab4582998b"
Nov 03 09:30:16.863068 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:16.863020    4473 scope.go:122] "RemoveContainer" containerID="e02f96b9634425dddf7f0c9656f78f28dcd43b7d6bdee30cedff5737039885d5"
Nov 03 09:30:16.863203 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:16.863187    4473 scope.go:122] "RemoveContainer" containerID="69ed0fad1f777d7ea9b7c683009246ca6b9bb1ece6f0af52bf32f01cda41e3db"
Nov 03 09:30:16.943388 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:30:16.943352    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:30:17.210331 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:17.210298    4473 generic.go:356] "Generic (PLEG): container finished" podID="9cce93be-a3e6-45db-9a17-3560f3822c2e" containerID="fc6cacad5619cc79e394867b8fe8a50378b20829082b6b3cbe08882e2c7c4e75" exitCode=0
Nov 03 09:30:17.210752 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:17.210361    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-465707/auth-test-model-465707-kserve-75c6b79fbf-4v284"
Nov 03 09:30:17.210752 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:17.210383    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-465707/auth-test-model-465707-kserve-75c6b79fbf-4v284" event={"ID":"9cce93be-a3e6-45db-9a17-3560f3822c2e","Type":"ContainerDied","Data":"fc6cacad5619cc79e394867b8fe8a50378b20829082b6b3cbe08882e2c7c4e75"}
Nov 03 09:30:17.210752 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:17.210422    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-465707/auth-test-model-465707-kserve-75c6b79fbf-4v284" event={"ID":"9cce93be-a3e6-45db-9a17-3560f3822c2e","Type":"ContainerDied","Data":"412eaf800ddcf710df84be3d693842673550dc0db9b4ed272bc35d3cacaaab17"}
Nov 03 09:30:17.210752 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:17.210445    4473 scope.go:122] "RemoveContainer" containerID="fc6cacad5619cc79e394867b8fe8a50378b20829082b6b3cbe08882e2c7c4e75"
Nov 03 09:30:17.212880 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:17.212855    4473 generic.go:356] "Generic (PLEG): container finished" podID="04abc6b1-3b23-40da-b7ec-5d341ff6ad14" containerID="bd192d3a2257535daadeea529c1c35667094e61d102fd3f6c5c5d8e918975800" exitCode=60
Nov 03 09:30:17.213025 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:17.212932    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894970-l595n" event={"ID":"04abc6b1-3b23-40da-b7ec-5d341ff6ad14","Type":"ContainerDied","Data":"bd192d3a2257535daadeea529c1c35667094e61d102fd3f6c5c5d8e918975800"}
Nov 03 09:30:17.213445 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:17.213425    4473 scope.go:122] "RemoveContainer" containerID="bd192d3a2257535daadeea529c1c35667094e61d102fd3f6c5c5d8e918975800"
Nov 03 09:30:17.213771 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:30:17.213748    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-77207e-29894970-l595n_odh-ai-gateway-infra(04abc6b1-3b23-40da-b7ec-5d341ff6ad14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894970-l595n" podUID="04abc6b1-3b23-40da-b7ec-5d341ff6ad14"
Nov 03 09:30:17.224690 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:17.224669    4473 scope.go:122] "RemoveContainer" containerID="fc6cacad5619cc79e394867b8fe8a50378b20829082b6b3cbe08882e2c7c4e75"
Nov 03 09:30:17.226410 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:30:17.226386    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc6cacad5619cc79e394867b8fe8a50378b20829082b6b3cbe08882e2c7c4e75\": container with ID starting with fc6cacad5619cc79e394867b8fe8a50378b20829082b6b3cbe08882e2c7c4e75 not found: ID does not exist" containerID="fc6cacad5619cc79e394867b8fe8a50378b20829082b6b3cbe08882e2c7c4e75"
Nov 03 09:30:17.226491 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:17.226420    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc6cacad5619cc79e394867b8fe8a50378b20829082b6b3cbe08882e2c7c4e75"} err="failed to get container status \"fc6cacad5619cc79e394867b8fe8a50378b20829082b6b3cbe08882e2c7c4e75\": rpc error: code = NotFound desc = could not find container \"fc6cacad5619cc79e394867b8fe8a50378b20829082b6b3cbe08882e2c7c4e75\": container with ID starting with fc6cacad5619cc79e394867b8fe8a50378b20829082b6b3cbe08882e2c7c4e75 not found: ID does not exist"
Nov 03 09:30:17.226491 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:17.226442    4473 scope.go:122] "RemoveContainer" containerID="91e030b5e320e1144e7e23fd8898b3039e4474abb85f97b7edaad0ced1fb17cb"
Nov 03 09:30:17.330734 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:17.330708    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894970-l595n"]
Nov 03 09:30:17.381229 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:17.381203    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-465707/auth-test-model-465707-kserve-75c6b79fbf-4v284"]
Nov 03 09:30:17.381338 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:17.381318    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894970-xq4mw"
Nov 03 09:30:17.386390 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:17.386339    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-465707/auth-test-model-465707-kserve-75c6b79fbf-4v284"]
Nov 03 09:30:17.435893 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:17.435865    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/475138bd-0332-4a16-b413-272ccf954d5a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/475138bd-0332-4a16-b413-272ccf954d5a-maas-api-service-ca\") pod \"475138bd-0332-4a16-b413-272ccf954d5a\" (UID: \"475138bd-0332-4a16-b413-272ccf954d5a\") "
Nov 03 09:30:17.436069 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:17.435926    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/475138bd-0332-4a16-b413-272ccf954d5a-kube-api-access-7slg4\" (UniqueName: \"kubernetes.io/projected/475138bd-0332-4a16-b413-272ccf954d5a-kube-api-access-7slg4\") pod \"475138bd-0332-4a16-b413-272ccf954d5a\" (UID: \"475138bd-0332-4a16-b413-272ccf954d5a\") "
Nov 03 09:30:17.436199 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:17.436177    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/475138bd-0332-4a16-b413-272ccf954d5a-maas-api-service-ca" pod "475138bd-0332-4a16-b413-272ccf954d5a" (UID: "475138bd-0332-4a16-b413-272ccf954d5a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:30:17.438172 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:17.438142    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/475138bd-0332-4a16-b413-272ccf954d5a-kube-api-access-7slg4" pod "475138bd-0332-4a16-b413-272ccf954d5a" (UID: "475138bd-0332-4a16-b413-272ccf954d5a"). InnerVolumeSpecName "kube-api-access-7slg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:30:17.537674 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:17.537640    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/475138bd-0332-4a16-b413-272ccf954d5a-maas-api-service-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:30:17.537839 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:17.537677    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7slg4\" (UniqueName: \"kubernetes.io/projected/475138bd-0332-4a16-b413-272ccf954d5a-kube-api-access-7slg4\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:30:17.863510 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:17.863481    4473 scope.go:122] "RemoveContainer" containerID="8eddf1f0d68391bbece2deb14412068347b7b8df14338ebc73bd3c53b517e2df"
Nov 03 09:30:17.863690 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:17.863674    4473 scope.go:122] "RemoveContainer" containerID="e4a8270e31f3a349cc7d95f6b313c029c2d9869906dd0da605dadd9f6dd311a6"
Nov 03 09:30:17.863904 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:17.863885    4473 scope.go:122] "RemoveContainer" containerID="aaea2e9b71699bf2b0e00222719b09a0e7f89b055579ed6723895ed55d12f698"
Nov 03 09:30:17.872062 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:17.872038    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cce93be-a3e6-45db-9a17-3560f3822c2e" path="/var/lib/kubelet/pods/9cce93be-a3e6-45db-9a17-3560f3822c2e/volumes"
Nov 03 09:30:18.235147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:18.235124    4473 generic.go:356] "Generic (PLEG): container finished" podID="d0f5712d-329f-4e4e-910f-2351f04f691e" containerID="0daf75d149b69622f926da8eb1abe51d4c6bccf062644f8bf120c6df2462b693" exitCode=60
Nov 03 09:30:18.235499 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:18.235201    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894970-f5nvf" event={"ID":"d0f5712d-329f-4e4e-910f-2351f04f691e","Type":"ContainerDied","Data":"0daf75d149b69622f926da8eb1abe51d4c6bccf062644f8bf120c6df2462b693"}
Nov 03 09:30:18.235499 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:18.235264    4473 scope.go:122] "RemoveContainer" containerID="08c62caa2b497c6dac2c7200e0d7bcc23f2e6b908a9c5cc3921e69ab4582998b"
Nov 03 09:30:18.235638 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:18.235596    4473 scope.go:122] "RemoveContainer" containerID="0daf75d149b69622f926da8eb1abe51d4c6bccf062644f8bf120c6df2462b693"
Nov 03 09:30:18.235914 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:30:18.235892    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894970-f5nvf_odh-ai-gateway-infra(d0f5712d-329f-4e4e-910f-2351f04f691e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894970-f5nvf" podUID="d0f5712d-329f-4e4e-910f-2351f04f691e"
Nov 03 09:30:18.236671 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:18.236656    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894970-xq4mw"
Nov 03 09:30:18.236772 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:18.236673    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894970-xq4mw" event={"ID":"475138bd-0332-4a16-b413-272ccf954d5a","Type":"ContainerDied","Data":"150f38d5acae6ae2a4ef468ee43e3851ba7fba3fb51263bb772e161a2259fc17"}
Nov 03 09:30:18.239906 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:18.239883    4473 generic.go:356] "Generic (PLEG): container finished" podID="ccf23ce8-844c-4924-bd1c-e90e1ea762ab" containerID="790638eca56fd1458d37007209e15b3155482f16f5175335a3d78f0d1e77efe9" exitCode=60
Nov 03 09:30:18.240019 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:18.239996    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894970-4g892" event={"ID":"ccf23ce8-844c-4924-bd1c-e90e1ea762ab","Type":"ContainerDied","Data":"790638eca56fd1458d37007209e15b3155482f16f5175335a3d78f0d1e77efe9"}
Nov 03 09:30:18.240430 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:18.240394    4473 scope.go:122] "RemoveContainer" containerID="790638eca56fd1458d37007209e15b3155482f16f5175335a3d78f0d1e77efe9"
Nov 03 09:30:18.240738 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:30:18.240708    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-465707-29894970-4g892_odh-ai-gateway-infra(ccf23ce8-844c-4924-bd1c-e90e1ea762ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894970-4g892" podUID="ccf23ce8-844c-4924-bd1c-e90e1ea762ab"
Nov 03 09:30:18.242426 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:18.242402    4473 generic.go:356] "Generic (PLEG): container finished" podID="b18d69e0-049e-4acd-bb6c-59299939e791" containerID="0c451ac6e583e95d0624b981d4e56a2f30dd90b6cd8a67536c789a0eefedc0bf" exitCode=60
Nov 03 09:30:18.242515 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:18.242503    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894970-swn74" event={"ID":"b18d69e0-049e-4acd-bb6c-59299939e791","Type":"ContainerDied","Data":"0c451ac6e583e95d0624b981d4e56a2f30dd90b6cd8a67536c789a0eefedc0bf"}
Nov 03 09:30:18.242807 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:18.242791    4473 scope.go:122] "RemoveContainer" containerID="0c451ac6e583e95d0624b981d4e56a2f30dd90b6cd8a67536c789a0eefedc0bf"
Nov 03 09:30:18.243127 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:30:18.243094    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894970-swn74_odh-ai-gateway-infra(b18d69e0-049e-4acd-bb6c-59299939e791)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894970-swn74" podUID="b18d69e0-049e-4acd-bb6c-59299939e791"
Nov 03 09:30:18.248679 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:18.248663    4473 scope.go:122] "RemoveContainer" containerID="58ff17c95ff6f02e5522f4d0f3977f6c498eea8a1b5917eb97c32dccc8436b61"
Nov 03 09:30:18.259093 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:18.258596    4473 scope.go:122] "RemoveContainer" containerID="69ed0fad1f777d7ea9b7c683009246ca6b9bb1ece6f0af52bf32f01cda41e3db"
Nov 03 09:30:18.283347 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:18.283326    4473 scope.go:122] "RemoveContainer" containerID="e02f96b9634425dddf7f0c9656f78f28dcd43b7d6bdee30cedff5737039885d5"
Nov 03 09:30:18.396285 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:18.396234    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894970-l595n"
Nov 03 09:30:18.450875 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:18.450839    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04abc6b1-3b23-40da-b7ec-5d341ff6ad14-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04abc6b1-3b23-40da-b7ec-5d341ff6ad14-maas-api-service-ca\") pod \"04abc6b1-3b23-40da-b7ec-5d341ff6ad14\" (UID: \"04abc6b1-3b23-40da-b7ec-5d341ff6ad14\") "
Nov 03 09:30:18.451078 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:18.450927    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04abc6b1-3b23-40da-b7ec-5d341ff6ad14-kube-api-access-8s8tl\" (UniqueName: \"kubernetes.io/projected/04abc6b1-3b23-40da-b7ec-5d341ff6ad14-kube-api-access-8s8tl\") pod \"04abc6b1-3b23-40da-b7ec-5d341ff6ad14\" (UID: \"04abc6b1-3b23-40da-b7ec-5d341ff6ad14\") "
Nov 03 09:30:18.452204 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:18.452173    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04abc6b1-3b23-40da-b7ec-5d341ff6ad14-maas-api-service-ca" pod "04abc6b1-3b23-40da-b7ec-5d341ff6ad14" (UID: "04abc6b1-3b23-40da-b7ec-5d341ff6ad14"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:30:18.459716 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:18.459681    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04abc6b1-3b23-40da-b7ec-5d341ff6ad14-kube-api-access-8s8tl" pod "04abc6b1-3b23-40da-b7ec-5d341ff6ad14" (UID: "04abc6b1-3b23-40da-b7ec-5d341ff6ad14"). InnerVolumeSpecName "kube-api-access-8s8tl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:30:18.552940 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:18.552901    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04abc6b1-3b23-40da-b7ec-5d341ff6ad14-maas-api-service-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:30:18.552940 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:18.552934    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s8tl\" (UniqueName: \"kubernetes.io/projected/04abc6b1-3b23-40da-b7ec-5d341ff6ad14-kube-api-access-8s8tl\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:30:18.599015 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:18.598976    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894970-xq4mw"]
Nov 03 09:30:18.601866 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:18.601845    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894970-xq4mw"]
Nov 03 09:30:18.863423 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:18.863392    4473 scope.go:122] "RemoveContainer" containerID="83f5de433e69f35bd5a24a8e0186bc25d8f444a8f156525ebe039e3043a760dc"
Nov 03 09:30:19.263184 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:19.263156    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894970-l595n"
Nov 03 09:30:19.263184 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:19.263167    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894970-l595n" event={"ID":"04abc6b1-3b23-40da-b7ec-5d341ff6ad14","Type":"ContainerDied","Data":"ed2b03dbb0d5e8efc29fadca18162a3f9660d4bb16b4d70f0821fd6f4001e520"}
Nov 03 09:30:19.263637 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:19.263251    4473 scope.go:122] "RemoveContainer" containerID="bd192d3a2257535daadeea529c1c35667094e61d102fd3f6c5c5d8e918975800"
Nov 03 09:30:19.265485 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:19.265461    4473 generic.go:356] "Generic (PLEG): container finished" podID="68e5001f-f741-4337-8767-8297949ac190" containerID="6512d3509f3d620d3fccc019698aac7e74b620344554fe48b3245c5de935a418" exitCode=60
Nov 03 09:30:19.265589 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:19.265538    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894970-4zzr8" event={"ID":"68e5001f-f741-4337-8767-8297949ac190","Type":"ContainerDied","Data":"6512d3509f3d620d3fccc019698aac7e74b620344554fe48b3245c5de935a418"}
Nov 03 09:30:19.265984 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:19.265935    4473 scope.go:122] "RemoveContainer" containerID="6512d3509f3d620d3fccc019698aac7e74b620344554fe48b3245c5de935a418"
Nov 03 09:30:19.266312 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:30:19.266256    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-433821-29894970-4zzr8_odh-ai-gateway-infra(68e5001f-f741-4337-8767-8297949ac190)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894970-4zzr8" podUID="68e5001f-f741-4337-8767-8297949ac190"
Nov 03 09:30:19.269826 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:19.269800    4473 generic.go:356] "Generic (PLEG): container finished" podID="b1a436ee-2304-4a63-9cd9-82e65871c169" containerID="2dcc835736c234b26792cc8914cc375ec6bdabd347f1fe559be7db82e779c996" exitCode=60
Nov 03 09:30:19.270046 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:19.269841    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894970-qgxzf" event={"ID":"b1a436ee-2304-4a63-9cd9-82e65871c169","Type":"ContainerDied","Data":"2dcc835736c234b26792cc8914cc375ec6bdabd347f1fe559be7db82e779c996"}
Nov 03 09:30:19.270476 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:19.270459    4473 scope.go:122] "RemoveContainer" containerID="2dcc835736c234b26792cc8914cc375ec6bdabd347f1fe559be7db82e779c996"
Nov 03 09:30:19.270678 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:30:19.270660    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894970-qgxzf_odh-ai-gateway-infra(b1a436ee-2304-4a63-9cd9-82e65871c169)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894970-qgxzf" podUID="b1a436ee-2304-4a63-9cd9-82e65871c169"
Nov 03 09:30:19.274994 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:19.274970    4473 generic.go:356] "Generic (PLEG): container finished" podID="7f59e915-c5e9-4ca4-b67f-fcdec7f83136" containerID="59fa6d29744142b0928d080362a51d1acffb67bb86373b3e7ec1354e5cdca0af" exitCode=60
Nov 03 09:30:19.275067 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:19.275007    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894970-lkt9m" event={"ID":"7f59e915-c5e9-4ca4-b67f-fcdec7f83136","Type":"ContainerDied","Data":"59fa6d29744142b0928d080362a51d1acffb67bb86373b3e7ec1354e5cdca0af"}
Nov 03 09:30:19.275374 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:19.275360    4473 scope.go:122] "RemoveContainer" containerID="59fa6d29744142b0928d080362a51d1acffb67bb86373b3e7ec1354e5cdca0af"
Nov 03 09:30:19.275624 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:30:19.275606    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894970-lkt9m_odh-ai-gateway-infra(7f59e915-c5e9-4ca4-b67f-fcdec7f83136)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894970-lkt9m" podUID="7f59e915-c5e9-4ca4-b67f-fcdec7f83136"
Nov 03 09:30:19.277031 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:19.277007    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894970-27x62" event={"ID":"5f055424-32fe-49ae-b6a9-4243eaf0c50b","Type":"ContainerStarted","Data":"4633c8a0aef3d46591d5050a5dece5bfe6ffb7d80743fbc52bb69446f02ac725"}
Nov 03 09:30:19.285090 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:19.285072    4473 scope.go:122] "RemoveContainer" containerID="e4a8270e31f3a349cc7d95f6b313c029c2d9869906dd0da605dadd9f6dd311a6"
Nov 03 09:30:19.297839 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:19.297821    4473 scope.go:122] "RemoveContainer" containerID="aaea2e9b71699bf2b0e00222719b09a0e7f89b055579ed6723895ed55d12f698"
Nov 03 09:30:19.311523 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:19.311509    4473 scope.go:122] "RemoveContainer" containerID="8eddf1f0d68391bbece2deb14412068347b7b8df14338ebc73bd3c53b517e2df"
Nov 03 09:30:19.364471 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:19.364449    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894970-f5nvf"]
Nov 03 09:30:19.503398 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:19.503372    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894970-f5nvf"
Nov 03 09:30:19.549071 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:19.549011    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894970-l595n"]
Nov 03 09:30:19.556646 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:19.556623    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894970-l595n"]
Nov 03 09:30:19.565012 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:19.564983    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0f5712d-329f-4e4e-910f-2351f04f691e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0f5712d-329f-4e4e-910f-2351f04f691e-maas-api-service-ca\") pod \"d0f5712d-329f-4e4e-910f-2351f04f691e\" (UID: \"d0f5712d-329f-4e4e-910f-2351f04f691e\") "
Nov 03 09:30:19.565096 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:19.565032    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0f5712d-329f-4e4e-910f-2351f04f691e-kube-api-access-blfh8\" (UniqueName: \"kubernetes.io/projected/d0f5712d-329f-4e4e-910f-2351f04f691e-kube-api-access-blfh8\") pod \"d0f5712d-329f-4e4e-910f-2351f04f691e\" (UID: \"d0f5712d-329f-4e4e-910f-2351f04f691e\") "
Nov 03 09:30:19.565406 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:19.565377    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0f5712d-329f-4e4e-910f-2351f04f691e-maas-api-service-ca" pod "d0f5712d-329f-4e4e-910f-2351f04f691e" (UID: "d0f5712d-329f-4e4e-910f-2351f04f691e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:30:19.565542 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:19.565527    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0f5712d-329f-4e4e-910f-2351f04f691e-maas-api-service-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:30:19.567243 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:19.567224    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0f5712d-329f-4e4e-910f-2351f04f691e-kube-api-access-blfh8" pod "d0f5712d-329f-4e4e-910f-2351f04f691e" (UID: "d0f5712d-329f-4e4e-910f-2351f04f691e"). InnerVolumeSpecName "kube-api-access-blfh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:30:19.667018 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:19.666992    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blfh8\" (UniqueName: \"kubernetes.io/projected/d0f5712d-329f-4e4e-910f-2351f04f691e-kube-api-access-blfh8\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:30:19.727810 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:19.727775    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894970-4g892"]
Nov 03 09:30:19.845744 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:19.845717    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894970-swn74"]
Nov 03 09:30:19.871971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:19.871896    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04abc6b1-3b23-40da-b7ec-5d341ff6ad14" path="/var/lib/kubelet/pods/04abc6b1-3b23-40da-b7ec-5d341ff6ad14/volumes"
Nov 03 09:30:19.872998 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:19.872975    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="475138bd-0332-4a16-b413-272ccf954d5a" path="/var/lib/kubelet/pods/475138bd-0332-4a16-b413-272ccf954d5a/volumes"
Nov 03 09:30:19.904419 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:19.904379    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894970-4g892"
Nov 03 09:30:19.970375 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:19.970336    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccf23ce8-844c-4924-bd1c-e90e1ea762ab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccf23ce8-844c-4924-bd1c-e90e1ea762ab-maas-api-service-ca\") pod \"ccf23ce8-844c-4924-bd1c-e90e1ea762ab\" (UID: \"ccf23ce8-844c-4924-bd1c-e90e1ea762ab\") "
Nov 03 09:30:19.970589 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:19.970469    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccf23ce8-844c-4924-bd1c-e90e1ea762ab-kube-api-access-2fg8r\" (UniqueName: \"kubernetes.io/projected/ccf23ce8-844c-4924-bd1c-e90e1ea762ab-kube-api-access-2fg8r\") pod \"ccf23ce8-844c-4924-bd1c-e90e1ea762ab\" (UID: \"ccf23ce8-844c-4924-bd1c-e90e1ea762ab\") "
Nov 03 09:30:19.971645 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:19.971616    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccf23ce8-844c-4924-bd1c-e90e1ea762ab-maas-api-service-ca" pod "ccf23ce8-844c-4924-bd1c-e90e1ea762ab" (UID: "ccf23ce8-844c-4924-bd1c-e90e1ea762ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:30:19.974054 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:19.974024    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccf23ce8-844c-4924-bd1c-e90e1ea762ab-kube-api-access-2fg8r" pod "ccf23ce8-844c-4924-bd1c-e90e1ea762ab" (UID: "ccf23ce8-844c-4924-bd1c-e90e1ea762ab"). InnerVolumeSpecName "kube-api-access-2fg8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:30:20.011054 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:20.011033    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894970-swn74"
Nov 03 09:30:20.071431 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:20.071359    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b18d69e0-049e-4acd-bb6c-59299939e791-kube-api-access-r6c7f\" (UniqueName: \"kubernetes.io/projected/b18d69e0-049e-4acd-bb6c-59299939e791-kube-api-access-r6c7f\") pod \"b18d69e0-049e-4acd-bb6c-59299939e791\" (UID: \"b18d69e0-049e-4acd-bb6c-59299939e791\") "
Nov 03 09:30:20.071431 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:20.071427    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b18d69e0-049e-4acd-bb6c-59299939e791-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b18d69e0-049e-4acd-bb6c-59299939e791-maas-api-service-ca\") pod \"b18d69e0-049e-4acd-bb6c-59299939e791\" (UID: \"b18d69e0-049e-4acd-bb6c-59299939e791\") "
Nov 03 09:30:20.071761 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:20.071747    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccf23ce8-844c-4924-bd1c-e90e1ea762ab-maas-api-service-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:30:20.071838 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:20.071763    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fg8r\" (UniqueName: \"kubernetes.io/projected/ccf23ce8-844c-4924-bd1c-e90e1ea762ab-kube-api-access-2fg8r\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:30:20.071838 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:20.071796    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b18d69e0-049e-4acd-bb6c-59299939e791-maas-api-service-ca" pod "b18d69e0-049e-4acd-bb6c-59299939e791" (UID: "b18d69e0-049e-4acd-bb6c-59299939e791"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:30:20.073594 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:20.073576    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b18d69e0-049e-4acd-bb6c-59299939e791-kube-api-access-r6c7f" pod "b18d69e0-049e-4acd-bb6c-59299939e791" (UID: "b18d69e0-049e-4acd-bb6c-59299939e791"). InnerVolumeSpecName "kube-api-access-r6c7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:30:20.173192 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:20.173158    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6c7f\" (UniqueName: \"kubernetes.io/projected/b18d69e0-049e-4acd-bb6c-59299939e791-kube-api-access-r6c7f\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:30:20.173192 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:20.173186    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b18d69e0-049e-4acd-bb6c-59299939e791-maas-api-service-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:30:20.284742 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:20.284714    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894970-f5nvf"
Nov 03 09:30:20.285176 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:20.284715    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894970-f5nvf" event={"ID":"d0f5712d-329f-4e4e-910f-2351f04f691e","Type":"ContainerDied","Data":"7feda9f6031ccc472e6b5d43b4be068a955afc47b0d76660d29dc25a2bad00eb"}
Nov 03 09:30:20.285176 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:20.284844    4473 scope.go:122] "RemoveContainer" containerID="0daf75d149b69622f926da8eb1abe51d4c6bccf062644f8bf120c6df2462b693"
Nov 03 09:30:20.288297 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:20.288271    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894970-4g892" event={"ID":"ccf23ce8-844c-4924-bd1c-e90e1ea762ab","Type":"ContainerDied","Data":"629168844efa47747a29067b27070c3675fe29203dc2092981e9aa7f5ec9d187"}
Nov 03 09:30:20.288419 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:20.288276    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894970-4g892"
Nov 03 09:30:20.292292 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:20.292259    4473 generic.go:356] "Generic (PLEG): container finished" podID="5f055424-32fe-49ae-b6a9-4243eaf0c50b" containerID="4633c8a0aef3d46591d5050a5dece5bfe6ffb7d80743fbc52bb69446f02ac725" exitCode=60
Nov 03 09:30:20.292413 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:20.292334    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894970-27x62" event={"ID":"5f055424-32fe-49ae-b6a9-4243eaf0c50b","Type":"ContainerDied","Data":"4633c8a0aef3d46591d5050a5dece5bfe6ffb7d80743fbc52bb69446f02ac725"}
Nov 03 09:30:20.292733 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:20.292714    4473 scope.go:122] "RemoveContainer" containerID="4633c8a0aef3d46591d5050a5dece5bfe6ffb7d80743fbc52bb69446f02ac725"
Nov 03 09:30:20.293003 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:30:20.292976    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894970-27x62_odh-ai-gateway-infra(5f055424-32fe-49ae-b6a9-4243eaf0c50b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894970-27x62" podUID="5f055424-32fe-49ae-b6a9-4243eaf0c50b"
Nov 03 09:30:20.299394 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:20.299375    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894970-swn74"
Nov 03 09:30:20.299485 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:20.299425    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894970-swn74" event={"ID":"b18d69e0-049e-4acd-bb6c-59299939e791","Type":"ContainerDied","Data":"a861637d747d6d5b2564735b442054e2782b206e2e78c8cde1c75ab11a21fa20"}
Nov 03 09:30:20.299872 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:20.299855    4473 scope.go:122] "RemoveContainer" containerID="790638eca56fd1458d37007209e15b3155482f16f5175335a3d78f0d1e77efe9"
Nov 03 09:30:20.309358 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:20.309344    4473 scope.go:122] "RemoveContainer" containerID="83f5de433e69f35bd5a24a8e0186bc25d8f444a8f156525ebe039e3043a760dc"
Nov 03 09:30:20.321813 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:20.321798    4473 scope.go:122] "RemoveContainer" containerID="0c451ac6e583e95d0624b981d4e56a2f30dd90b6cd8a67536c789a0eefedc0bf"
Nov 03 09:30:20.444752 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:20.444722    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894970-27x62"]
Nov 03 09:30:20.488633 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:20.488602    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894970-swn74"]
Nov 03 09:30:20.498249 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:20.498223    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-627e5f-29894970-swn74"]
Nov 03 09:30:20.609160 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:20.609088    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894970-f5nvf"]
Nov 03 09:30:20.617312 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:20.617288    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894970-f5nvf"]
Nov 03 09:30:20.683518 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:20.683485    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894970-qgxzf"]
Nov 03 09:30:20.805808 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:20.805752    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894970-lkt9m"]
Nov 03 09:30:20.859882 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:20.859811    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894970-4g892"]
Nov 03 09:30:20.870042 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:20.870011    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894970-4g892"]
Nov 03 09:30:20.883602 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:20.883581    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894970-qgxzf"
Nov 03 09:30:20.922081 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:20.922048    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894970-4zzr8"]
Nov 03 09:30:20.983269 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:20.983227    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1a436ee-2304-4a63-9cd9-82e65871c169-kube-api-access-6b9x5\" (UniqueName: \"kubernetes.io/projected/b1a436ee-2304-4a63-9cd9-82e65871c169-kube-api-access-6b9x5\") pod \"b1a436ee-2304-4a63-9cd9-82e65871c169\" (UID: \"b1a436ee-2304-4a63-9cd9-82e65871c169\") "
Nov 03 09:30:20.983769 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:20.983742    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1a436ee-2304-4a63-9cd9-82e65871c169-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a436ee-2304-4a63-9cd9-82e65871c169-maas-api-service-ca\") pod \"b1a436ee-2304-4a63-9cd9-82e65871c169\" (UID: \"b1a436ee-2304-4a63-9cd9-82e65871c169\") "
Nov 03 09:30:20.986077 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:20.986050    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1a436ee-2304-4a63-9cd9-82e65871c169-maas-api-service-ca" pod "b1a436ee-2304-4a63-9cd9-82e65871c169" (UID: "b1a436ee-2304-4a63-9cd9-82e65871c169"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:30:21.026571 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.026534    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1a436ee-2304-4a63-9cd9-82e65871c169-kube-api-access-6b9x5" pod "b1a436ee-2304-4a63-9cd9-82e65871c169" (UID: "b1a436ee-2304-4a63-9cd9-82e65871c169"). InnerVolumeSpecName "kube-api-access-6b9x5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:30:21.027718 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.027637    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894970-lkt9m"
Nov 03 09:30:21.088820 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.085216    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f59e915-c5e9-4ca4-b67f-fcdec7f83136-kube-api-access-bc8k5\" (UniqueName: \"kubernetes.io/projected/7f59e915-c5e9-4ca4-b67f-fcdec7f83136-kube-api-access-bc8k5\") pod \"7f59e915-c5e9-4ca4-b67f-fcdec7f83136\" (UID: \"7f59e915-c5e9-4ca4-b67f-fcdec7f83136\") "
Nov 03 09:30:21.088820 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.085275    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f59e915-c5e9-4ca4-b67f-fcdec7f83136-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f59e915-c5e9-4ca4-b67f-fcdec7f83136-maas-api-service-ca\") pod \"7f59e915-c5e9-4ca4-b67f-fcdec7f83136\" (UID: \"7f59e915-c5e9-4ca4-b67f-fcdec7f83136\") "
Nov 03 09:30:21.088820 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.085848    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b9x5\" (UniqueName: \"kubernetes.io/projected/b1a436ee-2304-4a63-9cd9-82e65871c169-kube-api-access-6b9x5\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:30:21.088820 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.085871    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1a436ee-2304-4a63-9cd9-82e65871c169-maas-api-service-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:30:21.088820 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.086357    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f59e915-c5e9-4ca4-b67f-fcdec7f83136-maas-api-service-ca" pod "7f59e915-c5e9-4ca4-b67f-fcdec7f83136" (UID: "7f59e915-c5e9-4ca4-b67f-fcdec7f83136"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:30:21.119171 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.117438    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f59e915-c5e9-4ca4-b67f-fcdec7f83136-kube-api-access-bc8k5" pod "7f59e915-c5e9-4ca4-b67f-fcdec7f83136" (UID: "7f59e915-c5e9-4ca4-b67f-fcdec7f83136"). InnerVolumeSpecName "kube-api-access-bc8k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:30:21.175061 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.175036    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894970-4zzr8"
Nov 03 09:30:21.186795 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.186768    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f59e915-c5e9-4ca4-b67f-fcdec7f83136-maas-api-service-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:30:21.186936 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.186798    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc8k5\" (UniqueName: \"kubernetes.io/projected/7f59e915-c5e9-4ca4-b67f-fcdec7f83136-kube-api-access-bc8k5\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:30:21.288045 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.288018    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68e5001f-f741-4337-8767-8297949ac190-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e5001f-f741-4337-8767-8297949ac190-maas-api-service-ca\") pod \"68e5001f-f741-4337-8767-8297949ac190\" (UID: \"68e5001f-f741-4337-8767-8297949ac190\") "
Nov 03 09:30:21.288481 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.288177    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68e5001f-f741-4337-8767-8297949ac190-kube-api-access-phbs2\" (UniqueName: \"kubernetes.io/projected/68e5001f-f741-4337-8767-8297949ac190-kube-api-access-phbs2\") pod \"68e5001f-f741-4337-8767-8297949ac190\" (UID: \"68e5001f-f741-4337-8767-8297949ac190\") "
Nov 03 09:30:21.288481 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.288440    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68e5001f-f741-4337-8767-8297949ac190-maas-api-service-ca" pod "68e5001f-f741-4337-8767-8297949ac190" (UID: "68e5001f-f741-4337-8767-8297949ac190"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:30:21.290512 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.290486    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68e5001f-f741-4337-8767-8297949ac190-kube-api-access-phbs2" pod "68e5001f-f741-4337-8767-8297949ac190" (UID: "68e5001f-f741-4337-8767-8297949ac190"). InnerVolumeSpecName "kube-api-access-phbs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:30:21.312753 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.312733    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894970-lkt9m"
Nov 03 09:30:21.312753 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.312731    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894970-lkt9m" event={"ID":"7f59e915-c5e9-4ca4-b67f-fcdec7f83136","Type":"ContainerDied","Data":"4fa1cccfc9532cf56f43ee39ed14813a7220deae1b5b54036cdab54995b98495"}
Nov 03 09:30:21.312959 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.312799    4473 scope.go:122] "RemoveContainer" containerID="59fa6d29744142b0928d080362a51d1acffb67bb86373b3e7ec1354e5cdca0af"
Nov 03 09:30:21.325720 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.325702    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894970-4zzr8"
Nov 03 09:30:21.325844 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.325732    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894970-4zzr8" event={"ID":"68e5001f-f741-4337-8767-8297949ac190","Type":"ContainerDied","Data":"39c4a140707de1d4629822fc93ac5216d0006aa4c9e63e4aece325982833535e"}
Nov 03 09:30:21.325844 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.325788    4473 scope.go:122] "RemoveContainer" containerID="6512d3509f3d620d3fccc019698aac7e74b620344554fe48b3245c5de935a418"
Nov 03 09:30:21.328613 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.328587    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894970-qgxzf" event={"ID":"b1a436ee-2304-4a63-9cd9-82e65871c169","Type":"ContainerDied","Data":"fcf9880f7742bc7457510b32bafd2fef0df96527ed7e1227f142f5809ae29369"}
Nov 03 09:30:21.328729 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.328679    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894970-qgxzf"
Nov 03 09:30:21.340782 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.340167    4473 scope.go:122] "RemoveContainer" containerID="2dcc835736c234b26792cc8914cc375ec6bdabd347f1fe559be7db82e779c996"
Nov 03 09:30:21.390052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.389799    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phbs2\" (UniqueName: \"kubernetes.io/projected/68e5001f-f741-4337-8767-8297949ac190-kube-api-access-phbs2\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:30:21.390052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.389825    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e5001f-f741-4337-8767-8297949ac190-maas-api-service-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:30:21.495659 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.495631    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894970-27x62"
Nov 03 09:30:21.591711 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.591681    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f055424-32fe-49ae-b6a9-4243eaf0c50b-kube-api-access-bwq44\" (UniqueName: \"kubernetes.io/projected/5f055424-32fe-49ae-b6a9-4243eaf0c50b-kube-api-access-bwq44\") pod \"5f055424-32fe-49ae-b6a9-4243eaf0c50b\" (UID: \"5f055424-32fe-49ae-b6a9-4243eaf0c50b\") "
Nov 03 09:30:21.591929 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.591911    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f055424-32fe-49ae-b6a9-4243eaf0c50b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f055424-32fe-49ae-b6a9-4243eaf0c50b-maas-api-service-ca\") pod \"5f055424-32fe-49ae-b6a9-4243eaf0c50b\" (UID: \"5f055424-32fe-49ae-b6a9-4243eaf0c50b\") "
Nov 03 09:30:21.592287 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.592262    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f055424-32fe-49ae-b6a9-4243eaf0c50b-maas-api-service-ca" pod "5f055424-32fe-49ae-b6a9-4243eaf0c50b" (UID: "5f055424-32fe-49ae-b6a9-4243eaf0c50b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:30:21.607288 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.607258    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f055424-32fe-49ae-b6a9-4243eaf0c50b-kube-api-access-bwq44" pod "5f055424-32fe-49ae-b6a9-4243eaf0c50b" (UID: "5f055424-32fe-49ae-b6a9-4243eaf0c50b"). InnerVolumeSpecName "kube-api-access-bwq44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:30:21.663303 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.663230    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894970-4zzr8"]
Nov 03 09:30:21.667905 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.667878    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-433821-29894970-4zzr8"]
Nov 03 09:30:21.693083 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.693052    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f055424-32fe-49ae-b6a9-4243eaf0c50b-maas-api-service-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:30:21.693217 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.693084    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwq44\" (UniqueName: \"kubernetes.io/projected/5f055424-32fe-49ae-b6a9-4243eaf0c50b-kube-api-access-bwq44\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:30:21.871071 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.871041    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68e5001f-f741-4337-8767-8297949ac190" path="/var/lib/kubelet/pods/68e5001f-f741-4337-8767-8297949ac190/volumes"
Nov 03 09:30:21.871475 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.871461    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b18d69e0-049e-4acd-bb6c-59299939e791" path="/var/lib/kubelet/pods/b18d69e0-049e-4acd-bb6c-59299939e791/volumes"
Nov 03 09:30:21.871815 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.871803    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccf23ce8-844c-4924-bd1c-e90e1ea762ab" path="/var/lib/kubelet/pods/ccf23ce8-844c-4924-bd1c-e90e1ea762ab/volumes"
Nov 03 09:30:21.872507 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.872491    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0f5712d-329f-4e4e-910f-2351f04f691e" path="/var/lib/kubelet/pods/d0f5712d-329f-4e4e-910f-2351f04f691e/volumes"
Nov 03 09:30:21.898012 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.897989    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894970-qgxzf"]
Nov 03 09:30:21.903591 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:21.903569    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894970-qgxzf"]
Nov 03 09:30:22.019542 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:22.019460    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894970-lkt9m"]
Nov 03 09:30:22.026722 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:22.026700    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894970-lkt9m"]
Nov 03 09:30:22.339093 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:22.339009    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894970-27x62"
Nov 03 09:30:22.339093 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:22.339063    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894970-27x62" event={"ID":"5f055424-32fe-49ae-b6a9-4243eaf0c50b","Type":"ContainerDied","Data":"c456acfe0f25258a54053255c3bb7ccf6133b643708657d0212eaa8769854aae"}
Nov 03 09:30:22.339624 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:22.339131    4473 scope.go:122] "RemoveContainer" containerID="4633c8a0aef3d46591d5050a5dece5bfe6ffb7d80743fbc52bb69446f02ac725"
Nov 03 09:30:22.511551 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:22.511505    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894970-27x62"]
Nov 03 09:30:22.514361 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:22.514337    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-98d5dd-29894970-27x62"]
Nov 03 09:30:23.870626 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:23.870597    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f055424-32fe-49ae-b6a9-4243eaf0c50b" path="/var/lib/kubelet/pods/5f055424-32fe-49ae-b6a9-4243eaf0c50b/volumes"
Nov 03 09:30:23.871038 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:23.871022    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f59e915-c5e9-4ca4-b67f-fcdec7f83136" path="/var/lib/kubelet/pods/7f59e915-c5e9-4ca4-b67f-fcdec7f83136/volumes"
Nov 03 09:30:23.871392 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:23.871380    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1a436ee-2304-4a63-9cd9-82e65871c169" path="/var/lib/kubelet/pods/b1a436ee-2304-4a63-9cd9-82e65871c169/volumes"
Nov 03 09:30:41.528795 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:41.528755    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b196c8/auth-test-model-b196c8-kserve-588d68f749-4t8w7"]
Nov 03 09:30:46.937158 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:30:46.937120    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:30:47.503338 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.503306    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f01a6d-55897df747-mvw5h"]
Nov 03 09:30:47.503811 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.503798    4473 cpu_manager.go:414] "containerMap: removing container" podUID="04abc6b1-3b23-40da-b7ec-5d341ff6ad14" containerName="cleanup"
Nov 03 09:30:47.503860 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.503813    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04abc6b1-3b23-40da-b7ec-5d341ff6ad14" containerName="cleanup"
Nov 03 09:30:47.503860 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.503823    4473 cpu_manager.go:414] "containerMap: removing container" podUID="68e5001f-f741-4337-8767-8297949ac190" containerName="cleanup"
Nov 03 09:30:47.503860 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.503827    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68e5001f-f741-4337-8767-8297949ac190" containerName="cleanup"
Nov 03 09:30:47.503860 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.503838    4473 cpu_manager.go:414] "containerMap: removing container" podUID="d0f5712d-329f-4e4e-910f-2351f04f691e" containerName="cleanup"
Nov 03 09:30:47.503860 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.503842    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0f5712d-329f-4e4e-910f-2351f04f691e" containerName="cleanup"
Nov 03 09:30:47.503860 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.503848    4473 cpu_manager.go:414] "containerMap: removing container" podUID="68e5001f-f741-4337-8767-8297949ac190" containerName="cleanup"
Nov 03 09:30:47.503860 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.503853    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68e5001f-f741-4337-8767-8297949ac190" containerName="cleanup"
Nov 03 09:30:47.503860 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.503860    4473 cpu_manager.go:414] "containerMap: removing container" podUID="5f055424-32fe-49ae-b6a9-4243eaf0c50b" containerName="cleanup"
Nov 03 09:30:47.504147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.503864    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f055424-32fe-49ae-b6a9-4243eaf0c50b" containerName="cleanup"
Nov 03 09:30:47.504147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.503870    4473 cpu_manager.go:414] "containerMap: removing container" podUID="ccf23ce8-844c-4924-bd1c-e90e1ea762ab" containerName="cleanup"
Nov 03 09:30:47.504147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.503875    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccf23ce8-844c-4924-bd1c-e90e1ea762ab" containerName="cleanup"
Nov 03 09:30:47.504147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.503882    4473 cpu_manager.go:414] "containerMap: removing container" podUID="5f055424-32fe-49ae-b6a9-4243eaf0c50b" containerName="cleanup"
Nov 03 09:30:47.504147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.503886    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f055424-32fe-49ae-b6a9-4243eaf0c50b" containerName="cleanup"
Nov 03 09:30:47.504147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.503894    4473 cpu_manager.go:414] "containerMap: removing container" podUID="d0f5712d-329f-4e4e-910f-2351f04f691e" containerName="cleanup"
Nov 03 09:30:47.504147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.503898    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0f5712d-329f-4e4e-910f-2351f04f691e" containerName="cleanup"
Nov 03 09:30:47.504147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.503904    4473 cpu_manager.go:414] "containerMap: removing container" podUID="475138bd-0332-4a16-b413-272ccf954d5a" containerName="cleanup"
Nov 03 09:30:47.504147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.503908    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="475138bd-0332-4a16-b413-272ccf954d5a" containerName="cleanup"
Nov 03 09:30:47.504147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.503915    4473 cpu_manager.go:414] "containerMap: removing container" podUID="b18d69e0-049e-4acd-bb6c-59299939e791" containerName="cleanup"
Nov 03 09:30:47.504147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.503922    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b18d69e0-049e-4acd-bb6c-59299939e791" containerName="cleanup"
Nov 03 09:30:47.504147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.503929    4473 cpu_manager.go:414] "containerMap: removing container" podUID="7f59e915-c5e9-4ca4-b67f-fcdec7f83136" containerName="cleanup"
Nov 03 09:30:47.504147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.503934    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f59e915-c5e9-4ca4-b67f-fcdec7f83136" containerName="cleanup"
Nov 03 09:30:47.504147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.503939    4473 cpu_manager.go:414] "containerMap: removing container" podUID="04abc6b1-3b23-40da-b7ec-5d341ff6ad14" containerName="cleanup"
Nov 03 09:30:47.504147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.503960    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04abc6b1-3b23-40da-b7ec-5d341ff6ad14" containerName="cleanup"
Nov 03 09:30:47.504147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.503971    4473 cpu_manager.go:414] "containerMap: removing container" podUID="04abc6b1-3b23-40da-b7ec-5d341ff6ad14" containerName="cleanup"
Nov 03 09:30:47.504147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.503977    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04abc6b1-3b23-40da-b7ec-5d341ff6ad14" containerName="cleanup"
Nov 03 09:30:47.504147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.503992    4473 cpu_manager.go:414] "containerMap: removing container" podUID="e99dc6a6-6192-473b-a7b2-27cc028c773c" containerName="main"
Nov 03 09:30:47.504147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.503999    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e99dc6a6-6192-473b-a7b2-27cc028c773c" containerName="main"
Nov 03 09:30:47.504147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504011    4473 cpu_manager.go:414] "containerMap: removing container" podUID="9cce93be-a3e6-45db-9a17-3560f3822c2e" containerName="main"
Nov 03 09:30:47.504147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504015    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cce93be-a3e6-45db-9a17-3560f3822c2e" containerName="main"
Nov 03 09:30:47.504147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504025    4473 cpu_manager.go:414] "containerMap: removing container" podUID="b1a436ee-2304-4a63-9cd9-82e65871c169" containerName="cleanup"
Nov 03 09:30:47.504147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504029    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1a436ee-2304-4a63-9cd9-82e65871c169" containerName="cleanup"
Nov 03 09:30:47.504147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504041    4473 cpu_manager.go:414] "containerMap: removing container" podUID="b18d69e0-049e-4acd-bb6c-59299939e791" containerName="cleanup"
Nov 03 09:30:47.504147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504045    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b18d69e0-049e-4acd-bb6c-59299939e791" containerName="cleanup"
Nov 03 09:30:47.504147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504053    4473 cpu_manager.go:414] "containerMap: removing container" podUID="475138bd-0332-4a16-b413-272ccf954d5a" containerName="cleanup"
Nov 03 09:30:47.504147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504058    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="475138bd-0332-4a16-b413-272ccf954d5a" containerName="cleanup"
Nov 03 09:30:47.504147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504064    4473 cpu_manager.go:414] "containerMap: removing container" podUID="7f59e915-c5e9-4ca4-b67f-fcdec7f83136" containerName="cleanup"
Nov 03 09:30:47.504147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504068    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f59e915-c5e9-4ca4-b67f-fcdec7f83136" containerName="cleanup"
Nov 03 09:30:47.504147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504073    4473 cpu_manager.go:414] "containerMap: removing container" podUID="7f59e915-c5e9-4ca4-b67f-fcdec7f83136" containerName="cleanup"
Nov 03 09:30:47.504147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504078    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f59e915-c5e9-4ca4-b67f-fcdec7f83136" containerName="cleanup"
Nov 03 09:30:47.504147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504093    4473 cpu_manager.go:414] "containerMap: removing container" podUID="b18d69e0-049e-4acd-bb6c-59299939e791" containerName="cleanup"
Nov 03 09:30:47.504147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504097    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b18d69e0-049e-4acd-bb6c-59299939e791" containerName="cleanup"
Nov 03 09:30:47.504147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504104    4473 cpu_manager.go:414] "containerMap: removing container" podUID="ccf23ce8-844c-4924-bd1c-e90e1ea762ab" containerName="cleanup"
Nov 03 09:30:47.504147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504108    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccf23ce8-844c-4924-bd1c-e90e1ea762ab" containerName="cleanup"
Nov 03 09:30:47.504147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504114    4473 cpu_manager.go:414] "containerMap: removing container" podUID="b1a436ee-2304-4a63-9cd9-82e65871c169" containerName="cleanup"
Nov 03 09:30:47.504147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504119    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1a436ee-2304-4a63-9cd9-82e65871c169" containerName="cleanup"
Nov 03 09:30:47.505294 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504214    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f59e915-c5e9-4ca4-b67f-fcdec7f83136" containerName="cleanup"
Nov 03 09:30:47.505294 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504221    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="04abc6b1-3b23-40da-b7ec-5d341ff6ad14" containerName="cleanup"
Nov 03 09:30:47.505294 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504228    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="68e5001f-f741-4337-8767-8297949ac190" containerName="cleanup"
Nov 03 09:30:47.505294 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504234    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="04abc6b1-3b23-40da-b7ec-5d341ff6ad14" containerName="cleanup"
Nov 03 09:30:47.505294 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504240    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f055424-32fe-49ae-b6a9-4243eaf0c50b" containerName="cleanup"
Nov 03 09:30:47.505294 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504246    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="e99dc6a6-6192-473b-a7b2-27cc028c773c" containerName="main"
Nov 03 09:30:47.505294 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504251    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cce93be-a3e6-45db-9a17-3560f3822c2e" containerName="main"
Nov 03 09:30:47.505294 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504257    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="475138bd-0332-4a16-b413-272ccf954d5a" containerName="cleanup"
Nov 03 09:30:47.505294 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504263    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1a436ee-2304-4a63-9cd9-82e65871c169" containerName="cleanup"
Nov 03 09:30:47.505294 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504269    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b18d69e0-049e-4acd-bb6c-59299939e791" containerName="cleanup"
Nov 03 09:30:47.505294 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504274    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="475138bd-0332-4a16-b413-272ccf954d5a" containerName="cleanup"
Nov 03 09:30:47.505294 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504279    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1a436ee-2304-4a63-9cd9-82e65871c169" containerName="cleanup"
Nov 03 09:30:47.505294 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504283    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f59e915-c5e9-4ca4-b67f-fcdec7f83136" containerName="cleanup"
Nov 03 09:30:47.505294 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504290    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccf23ce8-844c-4924-bd1c-e90e1ea762ab" containerName="cleanup"
Nov 03 09:30:47.505294 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504295    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b18d69e0-049e-4acd-bb6c-59299939e791" containerName="cleanup"
Nov 03 09:30:47.505294 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504299    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccf23ce8-844c-4924-bd1c-e90e1ea762ab" containerName="cleanup"
Nov 03 09:30:47.505294 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504307    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0f5712d-329f-4e4e-910f-2351f04f691e" containerName="cleanup"
Nov 03 09:30:47.505294 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504312    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="68e5001f-f741-4337-8767-8297949ac190" containerName="cleanup"
Nov 03 09:30:47.505294 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504317    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f055424-32fe-49ae-b6a9-4243eaf0c50b" containerName="cleanup"
Nov 03 09:30:47.505294 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504321    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0f5712d-329f-4e4e-910f-2351f04f691e" containerName="cleanup"
Nov 03 09:30:47.505294 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504325    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="475138bd-0332-4a16-b413-272ccf954d5a" containerName="cleanup"
Nov 03 09:30:47.505294 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.504331    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0f5712d-329f-4e4e-910f-2351f04f691e" containerName="cleanup"
Nov 03 09:30:47.509230 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.509210    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f01a6d-55897df747-mvw5h"
Nov 03 09:30:47.564661 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.564634    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f01a6d-55897df747-mvw5h"]
Nov 03 09:30:47.608002 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.607889    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdk4g\" (UniqueName: \"kubernetes.io/projected/c815db09-daba-4d3b-88f2-287d82d328f5-kube-api-access-hdk4g\") pod \"payload-pre-processing-e2e-extmod-a-f01a6d-55897df747-mvw5h\" (UID: \"c815db09-daba-4d3b-88f2-287d82d328f5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f01a6d-55897df747-mvw5h"
Nov 03 09:30:47.608248 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.608153    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c815db09-daba-4d3b-88f2-287d82d328f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-f01a6d-55897df747-mvw5h\" (UID: \"c815db09-daba-4d3b-88f2-287d82d328f5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f01a6d-55897df747-mvw5h"
Nov 03 09:30:47.611434 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.611409    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-f01a6d\""
Nov 03 09:30:47.611567 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.611545    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-f01a6d-dockercfg-rwxgz\""
Nov 03 09:30:47.708789 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.708759    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdk4g\" (UniqueName: \"kubernetes.io/projected/c815db09-daba-4d3b-88f2-287d82d328f5-kube-api-access-hdk4g\") pod \"payload-pre-processing-e2e-extmod-a-f01a6d-55897df747-mvw5h\" (UID: \"c815db09-daba-4d3b-88f2-287d82d328f5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f01a6d-55897df747-mvw5h"
Nov 03 09:30:47.709012 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.708891    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c815db09-daba-4d3b-88f2-287d82d328f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-f01a6d-55897df747-mvw5h\" (UID: \"c815db09-daba-4d3b-88f2-287d82d328f5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f01a6d-55897df747-mvw5h"
Nov 03 09:30:47.709611 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.709585    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c815db09-daba-4d3b-88f2-287d82d328f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-f01a6d-55897df747-mvw5h\" (UID: \"c815db09-daba-4d3b-88f2-287d82d328f5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f01a6d-55897df747-mvw5h"
Nov 03 09:30:47.768512 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.768443    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdk4g\" (UniqueName: \"kubernetes.io/projected/c815db09-daba-4d3b-88f2-287d82d328f5-kube-api-access-hdk4g\") pod \"payload-pre-processing-e2e-extmod-a-f01a6d-55897df747-mvw5h\" (UID: \"c815db09-daba-4d3b-88f2-287d82d328f5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f01a6d-55897df747-mvw5h"
Nov 03 09:30:47.823342 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:47.823319    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f01a6d-55897df747-mvw5h"
Nov 03 09:30:48.201048 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:48.201021    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f01a6d-55897df747-mvw5h"]
Nov 03 09:30:48.202412 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:30:48.202381    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc815db09_daba_4d3b_88f2_287d82d328f5.slice/crio-21f5e8c9c382bdc0c526e102bb6dfdb4069de5a491784f257d89700bdd952944 WatchSource:0}: Error finding container 21f5e8c9c382bdc0c526e102bb6dfdb4069de5a491784f257d89700bdd952944: Status 404 returned error can't find the container with id 21f5e8c9c382bdc0c526e102bb6dfdb4069de5a491784f257d89700bdd952944
Nov 03 09:30:48.523483 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:48.523393    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f01a6d-55897df747-mvw5h" event={"ID":"c815db09-daba-4d3b-88f2-287d82d328f5","Type":"ContainerStarted","Data":"9652c2bd1d727ee7f5fce7ea1b7de85be57f2c4e147e1ea9c77875bb956ada2a"}
Nov 03 09:30:48.523483 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:48.523455    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f01a6d-55897df747-mvw5h" event={"ID":"c815db09-daba-4d3b-88f2-287d82d328f5","Type":"ContainerStarted","Data":"21f5e8c9c382bdc0c526e102bb6dfdb4069de5a491784f257d89700bdd952944"}
Nov 03 09:30:48.523675 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:48.523486    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f01a6d-55897df747-mvw5h"
Nov 03 09:30:48.642628 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:48.642568    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f01a6d-55897df747-mvw5h" podStartSLOduration=1.642554599 podStartE2EDuration="1.642554599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:30:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:30:48.639253528 +0000 UTC m=+3072.908316981" watchObservedRunningTime="2026-11-03 09:30:48.642554599 +0000 UTC m=+3072.911618034"
Nov 03 09:30:56.281649 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:56.281614    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-3df1cc/auth-test-model-3df1cc-kserve-74596bcc89-6xm2k"]
Nov 03 09:30:56.558787 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:56.558695    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-b196c8/auth-test-model-b196c8-kserve-588d68f749-4t8w7" podUID="c7ce6825-d6c3-4d53-8759-0c0fb34b1f46" containerName="main" containerID="cri-o://435f3d0e9c1f2eec439cd11e69f78e4d10566507edc22645804eed51c221cba4" gracePeriod=45
Nov 03 09:30:56.587000 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:56.586940    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:30:56.830467 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:56.830443    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b196c8/auth-test-model-b196c8-kserve-588d68f749-4t8w7"
Nov 03 09:30:56.916988 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:56.916931    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-dshm\") pod \"c7ce6825-d6c3-4d53-8759-0c0fb34b1f46\" (UID: \"c7ce6825-d6c3-4d53-8759-0c0fb34b1f46\") "
Nov 03 09:30:56.917168 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:56.917042    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-home\" (UniqueName: \"kubernetes.io/empty-dir/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-home\") pod \"c7ce6825-d6c3-4d53-8759-0c0fb34b1f46\" (UID: \"c7ce6825-d6c3-4d53-8759-0c0fb34b1f46\") "
Nov 03 09:30:56.917168 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:56.917064    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-kube-api-access-bbkld\" (UniqueName: \"kubernetes.io/projected/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-kube-api-access-bbkld\") pod \"c7ce6825-d6c3-4d53-8759-0c0fb34b1f46\" (UID: \"c7ce6825-d6c3-4d53-8759-0c0fb34b1f46\") "
Nov 03 09:30:56.917168 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:56.917091    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-tmp-dir\") pod \"c7ce6825-d6c3-4d53-8759-0c0fb34b1f46\" (UID: \"c7ce6825-d6c3-4d53-8759-0c0fb34b1f46\") "
Nov 03 09:30:56.917168 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:56.917141    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-tls-certs\" (UniqueName: \"kubernetes.io/secret/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-tls-certs\") pod \"c7ce6825-d6c3-4d53-8759-0c0fb34b1f46\" (UID: \"c7ce6825-d6c3-4d53-8759-0c0fb34b1f46\") "
Nov 03 09:30:56.917396 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:56.917195    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-model-cache\") pod \"c7ce6825-d6c3-4d53-8759-0c0fb34b1f46\" (UID: \"c7ce6825-d6c3-4d53-8759-0c0fb34b1f46\") "
Nov 03 09:30:56.917396 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:56.917340    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-home" pod "c7ce6825-d6c3-4d53-8759-0c0fb34b1f46" (UID: "c7ce6825-d6c3-4d53-8759-0c0fb34b1f46"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:30:56.917396 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:56.917355    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-tmp-dir" pod "c7ce6825-d6c3-4d53-8759-0c0fb34b1f46" (UID: "c7ce6825-d6c3-4d53-8759-0c0fb34b1f46"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:30:56.917625 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:56.917605    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-model-cache" pod "c7ce6825-d6c3-4d53-8759-0c0fb34b1f46" (UID: "c7ce6825-d6c3-4d53-8759-0c0fb34b1f46"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:30:56.917704 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:56.917671    4473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-home\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:30:56.917704 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:56.917685    4473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-tmp-dir\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:30:56.919509 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:56.919484    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-kube-api-access-bbkld" pod "c7ce6825-d6c3-4d53-8759-0c0fb34b1f46" (UID: "c7ce6825-d6c3-4d53-8759-0c0fb34b1f46"). InnerVolumeSpecName "kube-api-access-bbkld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:30:56.919838 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:56.919819    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-tls-certs" pod "c7ce6825-d6c3-4d53-8759-0c0fb34b1f46" (UID: "c7ce6825-d6c3-4d53-8759-0c0fb34b1f46"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:30:56.919838 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:56.919825    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-dshm" pod "c7ce6825-d6c3-4d53-8759-0c0fb34b1f46" (UID: "c7ce6825-d6c3-4d53-8759-0c0fb34b1f46"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:30:57.019207 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:57.019171    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbkld\" (UniqueName: \"kubernetes.io/projected/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-kube-api-access-bbkld\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:30:57.019207 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:57.019205    4473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-tls-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:30:57.019458 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:57.019221    4473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-model-cache\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:30:57.019458 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:57.019235    4473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46-dshm\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:30:57.589724 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:57.589687    4473 generic.go:356] "Generic (PLEG): container finished" podID="c7ce6825-d6c3-4d53-8759-0c0fb34b1f46" containerID="435f3d0e9c1f2eec439cd11e69f78e4d10566507edc22645804eed51c221cba4" exitCode=0
Nov 03 09:30:57.589724 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:57.589726    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b196c8/auth-test-model-b196c8-kserve-588d68f749-4t8w7"
Nov 03 09:30:57.590301 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:57.589729    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b196c8/auth-test-model-b196c8-kserve-588d68f749-4t8w7" event={"ID":"c7ce6825-d6c3-4d53-8759-0c0fb34b1f46","Type":"ContainerDied","Data":"435f3d0e9c1f2eec439cd11e69f78e4d10566507edc22645804eed51c221cba4"}
Nov 03 09:30:57.590301 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:57.589769    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b196c8/auth-test-model-b196c8-kserve-588d68f749-4t8w7" event={"ID":"c7ce6825-d6c3-4d53-8759-0c0fb34b1f46","Type":"ContainerDied","Data":"f01731cd5513a1c387dfb30db4d988d949df4a08e21619ed2cc5cb7efa7ed30d"}
Nov 03 09:30:57.590301 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:57.589786    4473 scope.go:122] "RemoveContainer" containerID="435f3d0e9c1f2eec439cd11e69f78e4d10566507edc22645804eed51c221cba4"
Nov 03 09:30:57.599922 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:57.599904    4473 scope.go:122] "RemoveContainer" containerID="435f3d0e9c1f2eec439cd11e69f78e4d10566507edc22645804eed51c221cba4"
Nov 03 09:30:57.600246 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:30:57.600228    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"435f3d0e9c1f2eec439cd11e69f78e4d10566507edc22645804eed51c221cba4\": container with ID starting with 435f3d0e9c1f2eec439cd11e69f78e4d10566507edc22645804eed51c221cba4 not found: ID does not exist" containerID="435f3d0e9c1f2eec439cd11e69f78e4d10566507edc22645804eed51c221cba4"
Nov 03 09:30:57.600297 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:57.600254    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"435f3d0e9c1f2eec439cd11e69f78e4d10566507edc22645804eed51c221cba4"} err="failed to get container status \"435f3d0e9c1f2eec439cd11e69f78e4d10566507edc22645804eed51c221cba4\": rpc error: code = NotFound desc = could not find container \"435f3d0e9c1f2eec439cd11e69f78e4d10566507edc22645804eed51c221cba4\": container with ID starting with 435f3d0e9c1f2eec439cd11e69f78e4d10566507edc22645804eed51c221cba4 not found: ID does not exist"
Nov 03 09:30:57.760160 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:57.760130    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b196c8/auth-test-model-b196c8-kserve-588d68f749-4t8w7"]
Nov 03 09:30:57.810906 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:57.810876    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b196c8/auth-test-model-b196c8-kserve-588d68f749-4t8w7"]
Nov 03 09:30:57.870803 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:57.870768    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7ce6825-d6c3-4d53-8759-0c0fb34b1f46" path="/var/lib/kubelet/pods/c7ce6825-d6c3-4d53-8759-0c0fb34b1f46/volumes"
Nov 03 09:30:59.546206 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:30:59.546176    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f01a6d-55897df747-mvw5h"
Nov 03 09:31:04.449472 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:04.449435    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b47771b-52jsh"]
Nov 03 09:31:04.449982 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:04.449919    4473 cpu_manager.go:414] "containerMap: removing container" podUID="c7ce6825-d6c3-4d53-8759-0c0fb34b1f46" containerName="main"
Nov 03 09:31:04.449982 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:04.449930    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7ce6825-d6c3-4d53-8759-0c0fb34b1f46" containerName="main"
Nov 03 09:31:04.449982 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:04.449968    4473 cpu_manager.go:414] "containerMap: removing container" podUID="ccf23ce8-844c-4924-bd1c-e90e1ea762ab" containerName="cleanup"
Nov 03 09:31:04.449982 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:04.449974    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccf23ce8-844c-4924-bd1c-e90e1ea762ab" containerName="cleanup"
Nov 03 09:31:04.449982 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:04.449982    4473 cpu_manager.go:414] "containerMap: removing container" podUID="b1a436ee-2304-4a63-9cd9-82e65871c169" containerName="cleanup"
Nov 03 09:31:04.450174 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:04.449989    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1a436ee-2304-4a63-9cd9-82e65871c169" containerName="cleanup"
Nov 03 09:31:04.450174 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:04.450005    4473 cpu_manager.go:414] "containerMap: removing container" podUID="68e5001f-f741-4337-8767-8297949ac190" containerName="cleanup"
Nov 03 09:31:04.450174 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:04.450010    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68e5001f-f741-4337-8767-8297949ac190" containerName="cleanup"
Nov 03 09:31:04.450174 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:04.450018    4473 cpu_manager.go:414] "containerMap: removing container" podUID="5f055424-32fe-49ae-b6a9-4243eaf0c50b" containerName="cleanup"
Nov 03 09:31:04.450174 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:04.450023    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f055424-32fe-49ae-b6a9-4243eaf0c50b" containerName="cleanup"
Nov 03 09:31:04.450174 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:04.450029    4473 cpu_manager.go:414] "containerMap: removing container" podUID="d0f5712d-329f-4e4e-910f-2351f04f691e" containerName="cleanup"
Nov 03 09:31:04.450174 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:04.450033    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0f5712d-329f-4e4e-910f-2351f04f691e" containerName="cleanup"
Nov 03 09:31:04.450174 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:04.450041    4473 cpu_manager.go:414] "containerMap: removing container" podUID="475138bd-0332-4a16-b413-272ccf954d5a" containerName="cleanup"
Nov 03 09:31:04.450174 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:04.450045    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="475138bd-0332-4a16-b413-272ccf954d5a" containerName="cleanup"
Nov 03 09:31:04.450174 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:04.450122    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f59e915-c5e9-4ca4-b67f-fcdec7f83136" containerName="cleanup"
Nov 03 09:31:04.450174 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:04.450134    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b18d69e0-049e-4acd-bb6c-59299939e791" containerName="cleanup"
Nov 03 09:31:04.450174 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:04.450140    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccf23ce8-844c-4924-bd1c-e90e1ea762ab" containerName="cleanup"
Nov 03 09:31:04.450174 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:04.450150    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="68e5001f-f741-4337-8767-8297949ac190" containerName="cleanup"
Nov 03 09:31:04.450174 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:04.450155    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="04abc6b1-3b23-40da-b7ec-5d341ff6ad14" containerName="cleanup"
Nov 03 09:31:04.450174 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:04.450162    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f055424-32fe-49ae-b6a9-4243eaf0c50b" containerName="cleanup"
Nov 03 09:31:04.450174 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:04.450169    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1a436ee-2304-4a63-9cd9-82e65871c169" containerName="cleanup"
Nov 03 09:31:04.450174 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:04.450177    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7ce6825-d6c3-4d53-8759-0c0fb34b1f46" containerName="main"
Nov 03 09:31:04.454487 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:04.454465    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b47771b-52jsh"
Nov 03 09:31:04.499687 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:04.499653    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e55c656f-62c0-46fd-a33b-362d89080a95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b47771b-52jsh\" (UID: \"e55c656f-62c0-46fd-a33b-362d89080a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b47771b-52jsh"
Nov 03 09:31:04.499839 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:04.499695    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e55c656f-62c0-46fd-a33b-362d89080a95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b47771b-52jsh\" (UID: \"e55c656f-62c0-46fd-a33b-362d89080a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b47771b-52jsh"
Nov 03 09:31:04.499839 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:04.499794    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rwmx\" (UniqueName: \"kubernetes.io/projected/e55c656f-62c0-46fd-a33b-362d89080a95-kube-api-access-7rwmx\") pod \"maas-api-revoke-sub-2b47771b-52jsh\" (UID: \"e55c656f-62c0-46fd-a33b-362d89080a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b47771b-52jsh"
Nov 03 09:31:04.510055 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:04.510028    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b47771b-52jsh"]
Nov 03 09:31:04.510192 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:04.510125    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:31:04.556834 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:04.556814    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:31:04.557118 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:04.557098    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:31:04.557222 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:04.557125    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:31:04.601148 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:04.601116    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e55c656f-62c0-46fd-a33b-362d89080a95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b47771b-52jsh\" (UID: \"e55c656f-62c0-46fd-a33b-362d89080a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b47771b-52jsh"
Nov 03 09:31:04.601278 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:04.601153    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e55c656f-62c0-46fd-a33b-362d89080a95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b47771b-52jsh\" (UID: \"e55c656f-62c0-46fd-a33b-362d89080a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b47771b-52jsh"
Nov 03 09:31:04.601278 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:04.601201    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rwmx\" (UniqueName: \"kubernetes.io/projected/e55c656f-62c0-46fd-a33b-362d89080a95-kube-api-access-7rwmx\") pod \"maas-api-revoke-sub-2b47771b-52jsh\" (UID: \"e55c656f-62c0-46fd-a33b-362d89080a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b47771b-52jsh"
Nov 03 09:31:04.601771 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:04.601750    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e55c656f-62c0-46fd-a33b-362d89080a95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b47771b-52jsh\" (UID: \"e55c656f-62c0-46fd-a33b-362d89080a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b47771b-52jsh"
Nov 03 09:31:04.603670 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:04.603649    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e55c656f-62c0-46fd-a33b-362d89080a95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b47771b-52jsh\" (UID: \"e55c656f-62c0-46fd-a33b-362d89080a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b47771b-52jsh"
Nov 03 09:31:04.660531 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:04.660507    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rwmx\" (UniqueName: \"kubernetes.io/projected/e55c656f-62c0-46fd-a33b-362d89080a95-kube-api-access-7rwmx\") pod \"maas-api-revoke-sub-2b47771b-52jsh\" (UID: \"e55c656f-62c0-46fd-a33b-362d89080a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b47771b-52jsh"
Nov 03 09:31:04.784569 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:04.784507    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b47771b-52jsh"
Nov 03 09:31:05.167386 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:05.167360    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b47771b-52jsh"]
Nov 03 09:31:05.168143 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:31:05.168118    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode55c656f_62c0_46fd_a33b_362d89080a95.slice/crio-05686e98e0a0b7b3dff2b6166aece811151b1175f2202b6b676918a5c888f24a WatchSource:0}: Error finding container 05686e98e0a0b7b3dff2b6166aece811151b1175f2202b6b676918a5c888f24a: Status 404 returned error can't find the container with id 05686e98e0a0b7b3dff2b6166aece811151b1175f2202b6b676918a5c888f24a
Nov 03 09:31:05.641413 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:05.641369    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b47771b-52jsh" event={"ID":"e55c656f-62c0-46fd-a33b-362d89080a95","Type":"ContainerStarted","Data":"76fe69b20cbafe93f69cacafd299318b5a138a922d621ed67dc7890511d05457"}
Nov 03 09:31:05.641838 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:05.641430    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b47771b-52jsh" event={"ID":"e55c656f-62c0-46fd-a33b-362d89080a95","Type":"ContainerStarted","Data":"05686e98e0a0b7b3dff2b6166aece811151b1175f2202b6b676918a5c888f24a"}
Nov 03 09:31:05.757441 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:05.757347    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b47771b-52jsh" podStartSLOduration=1.757333709 podStartE2EDuration="1.757333709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:31:05.757099219 +0000 UTC m=+3090.026162653" watchObservedRunningTime="2026-11-03 09:31:05.757333709 +0000 UTC m=+3090.026397143"
Nov 03 09:31:06.649074 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:06.649037    4473 generic.go:356] "Generic (PLEG): container finished" podID="e55c656f-62c0-46fd-a33b-362d89080a95" containerID="76fe69b20cbafe93f69cacafd299318b5a138a922d621ed67dc7890511d05457" exitCode=0
Nov 03 09:31:06.649513 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:06.649112    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b47771b-52jsh" event={"ID":"e55c656f-62c0-46fd-a33b-362d89080a95","Type":"ContainerDied","Data":"76fe69b20cbafe93f69cacafd299318b5a138a922d621ed67dc7890511d05457"}
Nov 03 09:31:07.800717 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:07.800684    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b47771b-52jsh"
Nov 03 09:31:07.935130 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:07.935054    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e55c656f-62c0-46fd-a33b-362d89080a95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e55c656f-62c0-46fd-a33b-362d89080a95-maas-api-xks-ca\") pod \"e55c656f-62c0-46fd-a33b-362d89080a95\" (UID: \"e55c656f-62c0-46fd-a33b-362d89080a95\") "
Nov 03 09:31:07.935278 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:07.935247    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e55c656f-62c0-46fd-a33b-362d89080a95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e55c656f-62c0-46fd-a33b-362d89080a95-maas-api-openshift-ca\") pod \"e55c656f-62c0-46fd-a33b-362d89080a95\" (UID: \"e55c656f-62c0-46fd-a33b-362d89080a95\") "
Nov 03 09:31:07.935335 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:07.935274    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e55c656f-62c0-46fd-a33b-362d89080a95-kube-api-access-7rwmx\" (UniqueName: \"kubernetes.io/projected/e55c656f-62c0-46fd-a33b-362d89080a95-kube-api-access-7rwmx\") pod \"e55c656f-62c0-46fd-a33b-362d89080a95\" (UID: \"e55c656f-62c0-46fd-a33b-362d89080a95\") "
Nov 03 09:31:07.935900 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:07.935610    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e55c656f-62c0-46fd-a33b-362d89080a95-maas-api-openshift-ca" pod "e55c656f-62c0-46fd-a33b-362d89080a95" (UID: "e55c656f-62c0-46fd-a33b-362d89080a95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:31:07.935900 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:07.935724    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e55c656f-62c0-46fd-a33b-362d89080a95-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:31:07.937289 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:07.937262    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e55c656f-62c0-46fd-a33b-362d89080a95-maas-api-xks-ca" pod "e55c656f-62c0-46fd-a33b-362d89080a95" (UID: "e55c656f-62c0-46fd-a33b-362d89080a95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:31:07.937853 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:07.937824    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e55c656f-62c0-46fd-a33b-362d89080a95-kube-api-access-7rwmx" pod "e55c656f-62c0-46fd-a33b-362d89080a95" (UID: "e55c656f-62c0-46fd-a33b-362d89080a95"). InnerVolumeSpecName "kube-api-access-7rwmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:31:08.037053 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:08.037024    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rwmx\" (UniqueName: \"kubernetes.io/projected/e55c656f-62c0-46fd-a33b-362d89080a95-kube-api-access-7rwmx\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:31:08.037053 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:08.037050    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e55c656f-62c0-46fd-a33b-362d89080a95-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:31:08.661786 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:08.661746    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b47771b-52jsh" event={"ID":"e55c656f-62c0-46fd-a33b-362d89080a95","Type":"ContainerDied","Data":"05686e98e0a0b7b3dff2b6166aece811151b1175f2202b6b676918a5c888f24a"}
Nov 03 09:31:08.661993 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:08.661794    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05686e98e0a0b7b3dff2b6166aece811151b1175f2202b6b676918a5c888f24a"
Nov 03 09:31:08.661993 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:08.661755    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b47771b-52jsh"
Nov 03 09:31:11.302136 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:11.302078    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-3df1cc/auth-test-model-3df1cc-kserve-74596bcc89-6xm2k" podUID="0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e" containerName="main" containerID="cri-o://fe61a7e63f74925d4131c1ccd6919223c40b4b17582c88d41febfade36083fd0" gracePeriod=45
Nov 03 09:31:11.306909 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:11.306767    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-3df1cc/auth-test-model-3df1cc-kserve-74596bcc89-6xm2k" podUID="0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.163:8000/ready\": dial tcp 10.133.0.163:8000: connect: connection refused"
Nov 03 09:31:11.561480 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:11.561425    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-3df1cc/auth-test-model-3df1cc-kserve-74596bcc89-6xm2k"
Nov 03 09:31:11.674775 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:11.674749    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-tls-certs\" (UniqueName: \"kubernetes.io/secret/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-tls-certs\") pod \"0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e\" (UID: \"0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e\") "
Nov 03 09:31:11.674931 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:11.674797    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-home\" (UniqueName: \"kubernetes.io/empty-dir/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-home\") pod \"0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e\" (UID: \"0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e\") "
Nov 03 09:31:11.674931 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:11.674824    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-model-cache\") pod \"0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e\" (UID: \"0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e\") "
Nov 03 09:31:11.675124 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:11.674935    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-kube-api-access-h4zf7\" (UniqueName: \"kubernetes.io/projected/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-kube-api-access-h4zf7\") pod \"0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e\" (UID: \"0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e\") "
Nov 03 09:31:11.675124 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:11.675070    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-tmp-dir\") pod \"0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e\" (UID: \"0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e\") "
Nov 03 09:31:11.675124 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:11.675102    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-model-cache" pod "0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e" (UID: "0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:31:11.675283 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:11.675170    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-dshm\") pod \"0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e\" (UID: \"0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e\") "
Nov 03 09:31:11.675464 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:11.675416    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-tmp-dir" pod "0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e" (UID: "0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:31:11.675627 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:11.675610    4473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-model-cache\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:31:11.675680 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:11.675633    4473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-tmp-dir\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:31:11.677860 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:11.677832    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-home" pod "0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e" (UID: "0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:31:11.678813 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:11.678700    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-tls-certs" pod "0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e" (UID: "0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:31:11.678813 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:11.678715    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-kube-api-access-h4zf7" pod "0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e" (UID: "0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e"). InnerVolumeSpecName "kube-api-access-h4zf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:31:11.678813 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:11.678791    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-dshm" pod "0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e" (UID: "0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:31:11.687862 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:11.687834    4473 generic.go:356] "Generic (PLEG): container finished" podID="0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e" containerID="fe61a7e63f74925d4131c1ccd6919223c40b4b17582c88d41febfade36083fd0" exitCode=0
Nov 03 09:31:11.688020 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:11.687897    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-3df1cc/auth-test-model-3df1cc-kserve-74596bcc89-6xm2k"
Nov 03 09:31:11.688697 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:11.688032    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-3df1cc/auth-test-model-3df1cc-kserve-74596bcc89-6xm2k" event={"ID":"0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e","Type":"ContainerDied","Data":"fe61a7e63f74925d4131c1ccd6919223c40b4b17582c88d41febfade36083fd0"}
Nov 03 09:31:11.688697 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:11.688075    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-3df1cc/auth-test-model-3df1cc-kserve-74596bcc89-6xm2k" event={"ID":"0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e","Type":"ContainerDied","Data":"46210343e157776b5ea6590303a01c01187d9f3824c9e1dc4468a0b8e0387a58"}
Nov 03 09:31:11.688697 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:11.688099    4473 scope.go:122] "RemoveContainer" containerID="fe61a7e63f74925d4131c1ccd6919223c40b4b17582c88d41febfade36083fd0"
Nov 03 09:31:11.705928 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:11.705911    4473 scope.go:122] "RemoveContainer" containerID="fe61a7e63f74925d4131c1ccd6919223c40b4b17582c88d41febfade36083fd0"
Nov 03 09:31:11.706217 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:31:11.706190    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe61a7e63f74925d4131c1ccd6919223c40b4b17582c88d41febfade36083fd0\": container with ID starting with fe61a7e63f74925d4131c1ccd6919223c40b4b17582c88d41febfade36083fd0 not found: ID does not exist" containerID="fe61a7e63f74925d4131c1ccd6919223c40b4b17582c88d41febfade36083fd0"
Nov 03 09:31:11.706278 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:11.706229    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe61a7e63f74925d4131c1ccd6919223c40b4b17582c88d41febfade36083fd0"} err="failed to get container status \"fe61a7e63f74925d4131c1ccd6919223c40b4b17582c88d41febfade36083fd0\": rpc error: code = NotFound desc = could not find container \"fe61a7e63f74925d4131c1ccd6919223c40b4b17582c88d41febfade36083fd0\": container with ID starting with fe61a7e63f74925d4131c1ccd6919223c40b4b17582c88d41febfade36083fd0 not found: ID does not exist"
Nov 03 09:31:11.776867 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:11.776846    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4zf7\" (UniqueName: \"kubernetes.io/projected/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-kube-api-access-h4zf7\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:31:11.776867 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:11.776868    4473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-dshm\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:31:11.777037 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:11.776880    4473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-tls-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:31:11.777037 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:11.776888    4473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e-home\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:31:11.858793 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:11.858726    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-3df1cc/auth-test-model-3df1cc-kserve-74596bcc89-6xm2k"]
Nov 03 09:31:11.871106 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:11.871084    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-3df1cc/auth-test-model-3df1cc-kserve-74596bcc89-6xm2k"]
Nov 03 09:31:13.870168 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:13.870139    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e" path="/var/lib/kubelet/pods/0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e/volumes"
Nov 03 09:31:15.211156 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:15.211127    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-486ceaf4-xwz8x"]
Nov 03 09:31:15.211820 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:15.211799    4473 cpu_manager.go:414] "containerMap: removing container" podUID="0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e" containerName="main"
Nov 03 09:31:15.211915 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:15.211823    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e" containerName="main"
Nov 03 09:31:15.211915 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:15.211839    4473 cpu_manager.go:414] "containerMap: removing container" podUID="e55c656f-62c0-46fd-a33b-362d89080a95" containerName="revoke-keys"
Nov 03 09:31:15.211915 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:15.211848    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e55c656f-62c0-46fd-a33b-362d89080a95" containerName="revoke-keys"
Nov 03 09:31:15.212124 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:15.212043    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b8be3ca-23d3-4edd-94fb-41ee0a7e4d9e" containerName="main"
Nov 03 09:31:15.212124 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:15.212059    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="e55c656f-62c0-46fd-a33b-362d89080a95" containerName="revoke-keys"
Nov 03 09:31:15.216900 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:15.216878    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-486ceaf4-xwz8x"
Nov 03 09:31:15.275013 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:15.273784    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-486ceaf4-xwz8x"]
Nov 03 09:31:15.314380 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:15.314349    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9772119f-bb0f-44fa-8277-211ff4c2d429-maas-api-xks-ca\") pod \"maas-api-revoke-sub-486ceaf4-xwz8x\" (UID: \"9772119f-bb0f-44fa-8277-211ff4c2d429\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-486ceaf4-xwz8x"
Nov 03 09:31:15.314533 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:15.314383    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxwt6\" (UniqueName: \"kubernetes.io/projected/9772119f-bb0f-44fa-8277-211ff4c2d429-kube-api-access-nxwt6\") pod \"maas-api-revoke-sub-486ceaf4-xwz8x\" (UID: \"9772119f-bb0f-44fa-8277-211ff4c2d429\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-486ceaf4-xwz8x"
Nov 03 09:31:15.314533 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:15.314511    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9772119f-bb0f-44fa-8277-211ff4c2d429-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-486ceaf4-xwz8x\" (UID: \"9772119f-bb0f-44fa-8277-211ff4c2d429\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-486ceaf4-xwz8x"
Nov 03 09:31:15.319151 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:15.319130    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:31:15.319234 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:15.319136    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:31:15.319234 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:15.319154    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:31:15.320953 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:15.320926    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:31:15.415627 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:15.415592    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9772119f-bb0f-44fa-8277-211ff4c2d429-maas-api-xks-ca\") pod \"maas-api-revoke-sub-486ceaf4-xwz8x\" (UID: \"9772119f-bb0f-44fa-8277-211ff4c2d429\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-486ceaf4-xwz8x"
Nov 03 09:31:15.415772 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:15.415643    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxwt6\" (UniqueName: \"kubernetes.io/projected/9772119f-bb0f-44fa-8277-211ff4c2d429-kube-api-access-nxwt6\") pod \"maas-api-revoke-sub-486ceaf4-xwz8x\" (UID: \"9772119f-bb0f-44fa-8277-211ff4c2d429\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-486ceaf4-xwz8x"
Nov 03 09:31:15.415772 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:15.415728    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9772119f-bb0f-44fa-8277-211ff4c2d429-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-486ceaf4-xwz8x\" (UID: \"9772119f-bb0f-44fa-8277-211ff4c2d429\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-486ceaf4-xwz8x"
Nov 03 09:31:15.416475 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:15.416435    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9772119f-bb0f-44fa-8277-211ff4c2d429-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-486ceaf4-xwz8x\" (UID: \"9772119f-bb0f-44fa-8277-211ff4c2d429\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-486ceaf4-xwz8x"
Nov 03 09:31:15.418333 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:15.418307    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9772119f-bb0f-44fa-8277-211ff4c2d429-maas-api-xks-ca\") pod \"maas-api-revoke-sub-486ceaf4-xwz8x\" (UID: \"9772119f-bb0f-44fa-8277-211ff4c2d429\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-486ceaf4-xwz8x"
Nov 03 09:31:15.475595 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:15.475535    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxwt6\" (UniqueName: \"kubernetes.io/projected/9772119f-bb0f-44fa-8277-211ff4c2d429-kube-api-access-nxwt6\") pod \"maas-api-revoke-sub-486ceaf4-xwz8x\" (UID: \"9772119f-bb0f-44fa-8277-211ff4c2d429\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-486ceaf4-xwz8x"
Nov 03 09:31:15.530511 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:15.530486    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-486ceaf4-xwz8x"
Nov 03 09:31:15.905523 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:15.905500    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-486ceaf4-xwz8x"]
Nov 03 09:31:15.907151 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:31:15.907124    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9772119f_bb0f_44fa_8277_211ff4c2d429.slice/crio-77680b19ad282606497859fd9a798e73ed974c44ee62b590b3448331d32a894d WatchSource:0}: Error finding container 77680b19ad282606497859fd9a798e73ed974c44ee62b590b3448331d32a894d: Status 404 returned error can't find the container with id 77680b19ad282606497859fd9a798e73ed974c44ee62b590b3448331d32a894d
Nov 03 09:31:16.722113 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:16.722075    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-486ceaf4-xwz8x" event={"ID":"9772119f-bb0f-44fa-8277-211ff4c2d429","Type":"ContainerStarted","Data":"c362c59d5cc356724d48183077cd1b1c57e30c7c21d63f678264859f3a854dd0"}
Nov 03 09:31:16.722507 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:16.722129    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-486ceaf4-xwz8x" event={"ID":"9772119f-bb0f-44fa-8277-211ff4c2d429","Type":"ContainerStarted","Data":"77680b19ad282606497859fd9a798e73ed974c44ee62b590b3448331d32a894d"}
Nov 03 09:31:16.840352 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:16.840302    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-486ceaf4-xwz8x" podStartSLOduration=1.840287692 podStartE2EDuration="1.840287692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:31:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:31:16.837398295 +0000 UTC m=+3101.106461729" watchObservedRunningTime="2026-11-03 09:31:16.840287692 +0000 UTC m=+3101.109351125"
Nov 03 09:31:16.937724 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:31:16.937646    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:31:17.730527 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:17.730489    4473 generic.go:356] "Generic (PLEG): container finished" podID="9772119f-bb0f-44fa-8277-211ff4c2d429" containerID="c362c59d5cc356724d48183077cd1b1c57e30c7c21d63f678264859f3a854dd0" exitCode=0
Nov 03 09:31:17.730965 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:17.730581    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-486ceaf4-xwz8x" event={"ID":"9772119f-bb0f-44fa-8277-211ff4c2d429","Type":"ContainerDied","Data":"c362c59d5cc356724d48183077cd1b1c57e30c7c21d63f678264859f3a854dd0"}
Nov 03 09:31:18.890490 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:18.890470    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-486ceaf4-xwz8x"
Nov 03 09:31:19.054501 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:19.054421    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9772119f-bb0f-44fa-8277-211ff4c2d429-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9772119f-bb0f-44fa-8277-211ff4c2d429-maas-api-xks-ca\") pod \"9772119f-bb0f-44fa-8277-211ff4c2d429\" (UID: \"9772119f-bb0f-44fa-8277-211ff4c2d429\") "
Nov 03 09:31:19.054501 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:19.054472    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9772119f-bb0f-44fa-8277-211ff4c2d429-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9772119f-bb0f-44fa-8277-211ff4c2d429-maas-api-openshift-ca\") pod \"9772119f-bb0f-44fa-8277-211ff4c2d429\" (UID: \"9772119f-bb0f-44fa-8277-211ff4c2d429\") "
Nov 03 09:31:19.054726 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:19.054520    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9772119f-bb0f-44fa-8277-211ff4c2d429-kube-api-access-nxwt6\" (UniqueName: \"kubernetes.io/projected/9772119f-bb0f-44fa-8277-211ff4c2d429-kube-api-access-nxwt6\") pod \"9772119f-bb0f-44fa-8277-211ff4c2d429\" (UID: \"9772119f-bb0f-44fa-8277-211ff4c2d429\") "
Nov 03 09:31:19.054897 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:19.054856    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9772119f-bb0f-44fa-8277-211ff4c2d429-maas-api-openshift-ca" pod "9772119f-bb0f-44fa-8277-211ff4c2d429" (UID: "9772119f-bb0f-44fa-8277-211ff4c2d429"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:31:19.056887 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:19.056849    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9772119f-bb0f-44fa-8277-211ff4c2d429-maas-api-xks-ca" pod "9772119f-bb0f-44fa-8277-211ff4c2d429" (UID: "9772119f-bb0f-44fa-8277-211ff4c2d429"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:31:19.057041 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:19.056863    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9772119f-bb0f-44fa-8277-211ff4c2d429-kube-api-access-nxwt6" pod "9772119f-bb0f-44fa-8277-211ff4c2d429" (UID: "9772119f-bb0f-44fa-8277-211ff4c2d429"). InnerVolumeSpecName "kube-api-access-nxwt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:31:19.156234 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:19.156208    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9772119f-bb0f-44fa-8277-211ff4c2d429-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:31:19.156234 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:19.156232    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxwt6\" (UniqueName: \"kubernetes.io/projected/9772119f-bb0f-44fa-8277-211ff4c2d429-kube-api-access-nxwt6\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:31:19.156393 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:19.156243    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9772119f-bb0f-44fa-8277-211ff4c2d429-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:31:19.743103 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:19.743071    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-486ceaf4-xwz8x"
Nov 03 09:31:19.743284 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:19.743101    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-486ceaf4-xwz8x" event={"ID":"9772119f-bb0f-44fa-8277-211ff4c2d429","Type":"ContainerDied","Data":"77680b19ad282606497859fd9a798e73ed974c44ee62b590b3448331d32a894d"}
Nov 03 09:31:19.743284 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:19.743167    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77680b19ad282606497859fd9a798e73ed974c44ee62b590b3448331d32a894d"
Nov 03 09:31:31.981079 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:31.981043    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:31:46.937569 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:31:46.937531    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:31:59.386032 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:31:59.385996    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-f9b517/auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w"]
Nov 03 09:32:00.331109 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:00.331068    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9ed110/auth-test-model-9ed110-kserve-76968d48b9-wlwcm"]
Nov 03 09:32:01.814796 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:01.814761    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"]
Nov 03 09:32:01.815322 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:01.815257    4473 cpu_manager.go:414] "containerMap: removing container" podUID="9772119f-bb0f-44fa-8277-211ff4c2d429" containerName="revoke-keys"
Nov 03 09:32:01.815322 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:01.815269    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9772119f-bb0f-44fa-8277-211ff4c2d429" containerName="revoke-keys"
Nov 03 09:32:01.815501 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:01.815371    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="9772119f-bb0f-44fa-8277-211ff4c2d429" containerName="revoke-keys"
Nov 03 09:32:01.820170 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:01.820148    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"
Nov 03 09:32:01.839523 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:01.839495    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-istio-envoy\") pod \"e2e-autores-85fdd1-openshift-default-586f5945f5-g94st\" (UID: \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\") " pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"
Nov 03 09:32:01.839651 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:01.839565    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-istio-token\") pod \"e2e-autores-85fdd1-openshift-default-586f5945f5-g94st\" (UID: \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\") " pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"
Nov 03 09:32:01.839651 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:01.839619    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-istio-podinfo\") pod \"e2e-autores-85fdd1-openshift-default-586f5945f5-g94st\" (UID: \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\") " pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"
Nov 03 09:32:01.839725 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:01.839665    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6w6h\" (UniqueName: \"kubernetes.io/projected/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-kube-api-access-q6w6h\") pod \"e2e-autores-85fdd1-openshift-default-586f5945f5-g94st\" (UID: \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\") " pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"
Nov 03 09:32:01.839765 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:01.839753    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-istio-data\") pod \"e2e-autores-85fdd1-openshift-default-586f5945f5-g94st\" (UID: \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\") " pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"
Nov 03 09:32:01.839804 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:01.839778    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-credential-socket\") pod \"e2e-autores-85fdd1-openshift-default-586f5945f5-g94st\" (UID: \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\") " pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"
Nov 03 09:32:01.839804 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:01.839794    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-istiod-ca-cert\") pod \"e2e-autores-85fdd1-openshift-default-586f5945f5-g94st\" (UID: \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\") " pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"
Nov 03 09:32:01.839901 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:01.839821    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-workload-certs\") pod \"e2e-autores-85fdd1-openshift-default-586f5945f5-g94st\" (UID: \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\") " pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"
Nov 03 09:32:01.839981 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:01.839934    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-workload-socket\") pod \"e2e-autores-85fdd1-openshift-default-586f5945f5-g94st\" (UID: \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\") " pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"
Nov 03 09:32:01.877679 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:01.877651    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-85fdd1-openshift-default-dockercfg-qx2dd\""
Nov 03 09:32:01.932086 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:01.932048    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"]
Nov 03 09:32:01.940918 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:01.940862    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-istio-envoy\") pod \"e2e-autores-85fdd1-openshift-default-586f5945f5-g94st\" (UID: \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\") " pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"
Nov 03 09:32:01.941232 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:01.941203    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-istio-token\") pod \"e2e-autores-85fdd1-openshift-default-586f5945f5-g94st\" (UID: \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\") " pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"
Nov 03 09:32:01.941365 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:01.941277    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-istio-podinfo\") pod \"e2e-autores-85fdd1-openshift-default-586f5945f5-g94st\" (UID: \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\") " pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"
Nov 03 09:32:01.941365 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:01.941348    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6w6h\" (UniqueName: \"kubernetes.io/projected/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-kube-api-access-q6w6h\") pod \"e2e-autores-85fdd1-openshift-default-586f5945f5-g94st\" (UID: \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\") " pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"
Nov 03 09:32:01.941494 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:01.941414    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-istio-data\") pod \"e2e-autores-85fdd1-openshift-default-586f5945f5-g94st\" (UID: \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\") " pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"
Nov 03 09:32:01.941494 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:01.941453    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-credential-socket\") pod \"e2e-autores-85fdd1-openshift-default-586f5945f5-g94st\" (UID: \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\") " pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"
Nov 03 09:32:01.941494 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:01.941476    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-istiod-ca-cert\") pod \"e2e-autores-85fdd1-openshift-default-586f5945f5-g94st\" (UID: \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\") " pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"
Nov 03 09:32:01.941665 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:01.941512    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-workload-certs\") pod \"e2e-autores-85fdd1-openshift-default-586f5945f5-g94st\" (UID: \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\") " pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"
Nov 03 09:32:01.941665 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:01.941545    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-workload-socket\") pod \"e2e-autores-85fdd1-openshift-default-586f5945f5-g94st\" (UID: \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\") " pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"
Nov 03 09:32:01.943487 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:01.941921    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-workload-socket\") pod \"e2e-autores-85fdd1-openshift-default-586f5945f5-g94st\" (UID: \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\") " pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"
Nov 03 09:32:01.943487 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:01.942231    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-workload-certs\") pod \"e2e-autores-85fdd1-openshift-default-586f5945f5-g94st\" (UID: \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\") " pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"
Nov 03 09:32:01.943487 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:01.942612    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-istio-data\") pod \"e2e-autores-85fdd1-openshift-default-586f5945f5-g94st\" (UID: \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\") " pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"
Nov 03 09:32:01.943487 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:01.942738    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-credential-socket\") pod \"e2e-autores-85fdd1-openshift-default-586f5945f5-g94st\" (UID: \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\") " pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"
Nov 03 09:32:01.943487 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:01.943411    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-istiod-ca-cert\") pod \"e2e-autores-85fdd1-openshift-default-586f5945f5-g94st\" (UID: \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\") " pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"
Nov 03 09:32:01.944150 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:01.944121    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-istio-envoy\") pod \"e2e-autores-85fdd1-openshift-default-586f5945f5-g94st\" (UID: \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\") " pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"
Nov 03 09:32:01.944748 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:01.944724    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-istio-podinfo\") pod \"e2e-autores-85fdd1-openshift-default-586f5945f5-g94st\" (UID: \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\") " pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"
Nov 03 09:32:02.004494 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:02.004426    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-istio-token\") pod \"e2e-autores-85fdd1-openshift-default-586f5945f5-g94st\" (UID: \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\") " pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"
Nov 03 09:32:02.007266 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:02.007238    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6w6h\" (UniqueName: \"kubernetes.io/projected/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-kube-api-access-q6w6h\") pod \"e2e-autores-85fdd1-openshift-default-586f5945f5-g94st\" (UID: \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\") " pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"
Nov 03 09:32:02.135932 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:02.135898    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"
Nov 03 09:32:02.582897 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:02.582215    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"]
Nov 03 09:32:02.589341 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:32:02.589312    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbf6f0a2_6f2d_40ce_801c_ff3830b0c75c.slice/crio-1f05277119fdf2560e86d3d15be519a87c6b2da770377f5383cef16b8a4cd2bb WatchSource:0}: Error finding container 1f05277119fdf2560e86d3d15be519a87c6b2da770377f5383cef16b8a4cd2bb: Status 404 returned error can't find the container with id 1f05277119fdf2560e86d3d15be519a87c6b2da770377f5383cef16b8a4cd2bb
Nov 03 09:32:02.592535 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:02.592497    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:32:02.592627 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:02.592566    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:32:02.592627 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:02.592595    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:32:03.016562 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:03.016519    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st" event={"ID":"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c","Type":"ContainerStarted","Data":"d560f6b2c2b63d16e7a40fa4597ec897b3ed2f64a9f7bf5193d6ea6701add8ca"}
Nov 03 09:32:03.017024 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:03.016596    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st" event={"ID":"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c","Type":"ContainerStarted","Data":"1f05277119fdf2560e86d3d15be519a87c6b2da770377f5383cef16b8a4cd2bb"}
Nov 03 09:32:03.136552 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:03.136519    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"
Nov 03 09:32:03.142342 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:03.142317    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"
Nov 03 09:32:03.146918 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:03.146866    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st" podStartSLOduration=2.146853366 podStartE2EDuration="2.146853366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:32:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:32:03.144615131 +0000 UTC m=+3147.413678564" watchObservedRunningTime="2026-11-03 09:32:03.146853366 +0000 UTC m=+3147.415916797"
Nov 03 09:32:04.022214 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:04.022181    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"
Nov 03 09:32:04.023335 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:04.023314    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"
Nov 03 09:32:14.415836 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:14.415773    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-f9b517/auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w" podUID="254e50ff-0d1c-48f6-83ca-a88d9effba4d" containerName="main" containerID="cri-o://ad2d583e863e7bf92a92c49dda008703032904d4e0a0373c6e89368cf0a64afe" gracePeriod=45
Nov 03 09:32:14.417917 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:14.417894    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:32:14.690276 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:14.690253    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-f9b517/auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w"
Nov 03 09:32:14.797380 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:14.797352    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/254e50ff-0d1c-48f6-83ca-a88d9effba4d-tls-certs\" (UniqueName: \"kubernetes.io/secret/254e50ff-0d1c-48f6-83ca-a88d9effba4d-tls-certs\") pod \"254e50ff-0d1c-48f6-83ca-a88d9effba4d\" (UID: \"254e50ff-0d1c-48f6-83ca-a88d9effba4d\") "
Nov 03 09:32:14.797594 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:14.797396    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/254e50ff-0d1c-48f6-83ca-a88d9effba4d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/254e50ff-0d1c-48f6-83ca-a88d9effba4d-dshm\") pod \"254e50ff-0d1c-48f6-83ca-a88d9effba4d\" (UID: \"254e50ff-0d1c-48f6-83ca-a88d9effba4d\") "
Nov 03 09:32:14.797594 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:14.797486    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/254e50ff-0d1c-48f6-83ca-a88d9effba4d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/254e50ff-0d1c-48f6-83ca-a88d9effba4d-tmp-dir\") pod \"254e50ff-0d1c-48f6-83ca-a88d9effba4d\" (UID: \"254e50ff-0d1c-48f6-83ca-a88d9effba4d\") "
Nov 03 09:32:14.797594 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:14.797531    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/254e50ff-0d1c-48f6-83ca-a88d9effba4d-home\" (UniqueName: \"kubernetes.io/empty-dir/254e50ff-0d1c-48f6-83ca-a88d9effba4d-home\") pod \"254e50ff-0d1c-48f6-83ca-a88d9effba4d\" (UID: \"254e50ff-0d1c-48f6-83ca-a88d9effba4d\") "
Nov 03 09:32:14.797783 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:14.797668    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/254e50ff-0d1c-48f6-83ca-a88d9effba4d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/254e50ff-0d1c-48f6-83ca-a88d9effba4d-model-cache\") pod \"254e50ff-0d1c-48f6-83ca-a88d9effba4d\" (UID: \"254e50ff-0d1c-48f6-83ca-a88d9effba4d\") "
Nov 03 09:32:14.797783 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:14.797712    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/254e50ff-0d1c-48f6-83ca-a88d9effba4d-kube-api-access-vbjvk\" (UniqueName: \"kubernetes.io/projected/254e50ff-0d1c-48f6-83ca-a88d9effba4d-kube-api-access-vbjvk\") pod \"254e50ff-0d1c-48f6-83ca-a88d9effba4d\" (UID: \"254e50ff-0d1c-48f6-83ca-a88d9effba4d\") "
Nov 03 09:32:14.797783 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:14.797747    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/254e50ff-0d1c-48f6-83ca-a88d9effba4d-tmp-dir" pod "254e50ff-0d1c-48f6-83ca-a88d9effba4d" (UID: "254e50ff-0d1c-48f6-83ca-a88d9effba4d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:32:14.797975 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:14.797855    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/254e50ff-0d1c-48f6-83ca-a88d9effba4d-home" pod "254e50ff-0d1c-48f6-83ca-a88d9effba4d" (UID: "254e50ff-0d1c-48f6-83ca-a88d9effba4d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:32:14.797975 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:14.797900    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/254e50ff-0d1c-48f6-83ca-a88d9effba4d-model-cache" pod "254e50ff-0d1c-48f6-83ca-a88d9effba4d" (UID: "254e50ff-0d1c-48f6-83ca-a88d9effba4d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:32:14.798301 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:14.798282    4473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/254e50ff-0d1c-48f6-83ca-a88d9effba4d-model-cache\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:32:14.798370 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:14.798305    4473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/254e50ff-0d1c-48f6-83ca-a88d9effba4d-tmp-dir\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:32:14.798370 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:14.798318    4473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/254e50ff-0d1c-48f6-83ca-a88d9effba4d-home\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:32:14.799870 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:14.799842    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/254e50ff-0d1c-48f6-83ca-a88d9effba4d-dshm" pod "254e50ff-0d1c-48f6-83ca-a88d9effba4d" (UID: "254e50ff-0d1c-48f6-83ca-a88d9effba4d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:32:14.800055 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:14.799885    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/254e50ff-0d1c-48f6-83ca-a88d9effba4d-tls-certs" pod "254e50ff-0d1c-48f6-83ca-a88d9effba4d" (UID: "254e50ff-0d1c-48f6-83ca-a88d9effba4d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:32:14.800131 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:14.800092    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/254e50ff-0d1c-48f6-83ca-a88d9effba4d-kube-api-access-vbjvk" pod "254e50ff-0d1c-48f6-83ca-a88d9effba4d" (UID: "254e50ff-0d1c-48f6-83ca-a88d9effba4d"). InnerVolumeSpecName "kube-api-access-vbjvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:32:14.899312 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:14.899278    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbjvk\" (UniqueName: \"kubernetes.io/projected/254e50ff-0d1c-48f6-83ca-a88d9effba4d-kube-api-access-vbjvk\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:32:14.899312 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:14.899305    4473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/254e50ff-0d1c-48f6-83ca-a88d9effba4d-tls-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:32:14.899312 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:14.899315    4473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/254e50ff-0d1c-48f6-83ca-a88d9effba4d-dshm\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:32:15.044711 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.044627    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-85fdd1-887588c8c-b9dsj"]
Nov 03 09:32:15.045195 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.045181    4473 cpu_manager.go:414] "containerMap: removing container" podUID="254e50ff-0d1c-48f6-83ca-a88d9effba4d" containerName="main"
Nov 03 09:32:15.045252 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.045197    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="254e50ff-0d1c-48f6-83ca-a88d9effba4d" containerName="main"
Nov 03 09:32:15.045332 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.045323    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="254e50ff-0d1c-48f6-83ca-a88d9effba4d" containerName="main"
Nov 03 09:32:15.048682 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.048663    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-85fdd1-887588c8c-b9dsj"
Nov 03 09:32:15.091791 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.091755    4473 generic.go:356] "Generic (PLEG): container finished" podID="254e50ff-0d1c-48f6-83ca-a88d9effba4d" containerID="ad2d583e863e7bf92a92c49dda008703032904d4e0a0373c6e89368cf0a64afe" exitCode=0
Nov 03 09:32:15.092014 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.091809    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-f9b517/auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w"
Nov 03 09:32:15.092014 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.091860    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-f9b517/auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w" event={"ID":"254e50ff-0d1c-48f6-83ca-a88d9effba4d","Type":"ContainerDied","Data":"ad2d583e863e7bf92a92c49dda008703032904d4e0a0373c6e89368cf0a64afe"}
Nov 03 09:32:15.092014 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.091910    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-f9b517/auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w" event={"ID":"254e50ff-0d1c-48f6-83ca-a88d9effba4d","Type":"ContainerDied","Data":"40ab3f4a94e6a2e8a06a25edb512e0da3dc7250ddba3322f3f27dadb79eec469"}
Nov 03 09:32:15.092014 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.091929    4473 scope.go:122] "RemoveContainer" containerID="ad2d583e863e7bf92a92c49dda008703032904d4e0a0373c6e89368cf0a64afe"
Nov 03 09:32:15.101708 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.101680    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxks7\" (UniqueName: \"kubernetes.io/projected/e71ebab7-6acf-4f99-bf01-735064ba50e9-kube-api-access-lxks7\") pod \"payload-pre-processing-e2e-autores-85fdd1-887588c8c-b9dsj\" (UID: \"e71ebab7-6acf-4f99-bf01-735064ba50e9\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-85fdd1-887588c8c-b9dsj"
Nov 03 09:32:15.101845 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.101825    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e71ebab7-6acf-4f99-bf01-735064ba50e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-85fdd1-887588c8c-b9dsj\" (UID: \"e71ebab7-6acf-4f99-bf01-735064ba50e9\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-85fdd1-887588c8c-b9dsj"
Nov 03 09:32:15.103688 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.103148    4473 scope.go:122] "RemoveContainer" containerID="ad2d583e863e7bf92a92c49dda008703032904d4e0a0373c6e89368cf0a64afe"
Nov 03 09:32:15.103688 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:32:15.103666    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad2d583e863e7bf92a92c49dda008703032904d4e0a0373c6e89368cf0a64afe\": container with ID starting with ad2d583e863e7bf92a92c49dda008703032904d4e0a0373c6e89368cf0a64afe not found: ID does not exist" containerID="ad2d583e863e7bf92a92c49dda008703032904d4e0a0373c6e89368cf0a64afe"
Nov 03 09:32:15.103840 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.103701    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad2d583e863e7bf92a92c49dda008703032904d4e0a0373c6e89368cf0a64afe"} err="failed to get container status \"ad2d583e863e7bf92a92c49dda008703032904d4e0a0373c6e89368cf0a64afe\": rpc error: code = NotFound desc = could not find container \"ad2d583e863e7bf92a92c49dda008703032904d4e0a0373c6e89368cf0a64afe\": container with ID starting with ad2d583e863e7bf92a92c49dda008703032904d4e0a0373c6e89368cf0a64afe not found: ID does not exist"
Nov 03 09:32:15.104548 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.104428    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-85fdd1-dockercfg-d9t5w\""
Nov 03 09:32:15.104548 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.104497    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-85fdd1\""
Nov 03 09:32:15.108525 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.108500    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-85fdd1-887588c8c-b9dsj"]
Nov 03 09:32:15.207797 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.207748    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxks7\" (UniqueName: \"kubernetes.io/projected/e71ebab7-6acf-4f99-bf01-735064ba50e9-kube-api-access-lxks7\") pod \"payload-pre-processing-e2e-autores-85fdd1-887588c8c-b9dsj\" (UID: \"e71ebab7-6acf-4f99-bf01-735064ba50e9\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-85fdd1-887588c8c-b9dsj"
Nov 03 09:32:15.208090 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.207921    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e71ebab7-6acf-4f99-bf01-735064ba50e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-85fdd1-887588c8c-b9dsj\" (UID: \"e71ebab7-6acf-4f99-bf01-735064ba50e9\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-85fdd1-887588c8c-b9dsj"
Nov 03 09:32:15.208750 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.208728    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e71ebab7-6acf-4f99-bf01-735064ba50e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-85fdd1-887588c8c-b9dsj\" (UID: \"e71ebab7-6acf-4f99-bf01-735064ba50e9\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-85fdd1-887588c8c-b9dsj"
Nov 03 09:32:15.218329 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.218304    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-85fdd1-64678c8bb5-hk2dj"]
Nov 03 09:32:15.224706 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.224675    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-85fdd1-64678c8bb5-hk2dj"
Nov 03 09:32:15.271240 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.271214    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxks7\" (UniqueName: \"kubernetes.io/projected/e71ebab7-6acf-4f99-bf01-735064ba50e9-kube-api-access-lxks7\") pod \"payload-pre-processing-e2e-autores-85fdd1-887588c8c-b9dsj\" (UID: \"e71ebab7-6acf-4f99-bf01-735064ba50e9\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-85fdd1-887588c8c-b9dsj"
Nov 03 09:32:15.280866 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.280836    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-85fdd1-64678c8bb5-hk2dj"]
Nov 03 09:32:15.291571 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.291547    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-f9b517/auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w"]
Nov 03 09:32:15.299398 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.299323    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-f9b517/auth-test-model-f9b517-kserve-669dbcf6bf-h7s7w"]
Nov 03 09:32:15.309407 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.309380    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/736bfa13-e012-429c-95ff-96b1e83245e6-plugins-config-volume\") pod \"payload-processing-e2e-autores-85fdd1-64678c8bb5-hk2dj\" (UID: \"736bfa13-e012-429c-95ff-96b1e83245e6\") " pod="openshift-ingress/payload-processing-e2e-autores-85fdd1-64678c8bb5-hk2dj"
Nov 03 09:32:15.309632 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.309438    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wt5n\" (UniqueName: \"kubernetes.io/projected/736bfa13-e012-429c-95ff-96b1e83245e6-kube-api-access-8wt5n\") pod \"payload-processing-e2e-autores-85fdd1-64678c8bb5-hk2dj\" (UID: \"736bfa13-e012-429c-95ff-96b1e83245e6\") " pod="openshift-ingress/payload-processing-e2e-autores-85fdd1-64678c8bb5-hk2dj"
Nov 03 09:32:15.359676 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.359615    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-9ed110/auth-test-model-9ed110-kserve-76968d48b9-wlwcm" podUID="9dc0565c-837f-432a-bf4e-3b0f611a58fa" containerName="main" containerID="cri-o://c5c8564f4150b8febd3bf9e71346d1f198d036cb7296607df6cae603f6e42bd7" gracePeriod=45
Nov 03 09:32:15.364040 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.363814    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-85fdd1-887588c8c-b9dsj"
Nov 03 09:32:15.410700 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.410541    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/736bfa13-e012-429c-95ff-96b1e83245e6-plugins-config-volume\") pod \"payload-processing-e2e-autores-85fdd1-64678c8bb5-hk2dj\" (UID: \"736bfa13-e012-429c-95ff-96b1e83245e6\") " pod="openshift-ingress/payload-processing-e2e-autores-85fdd1-64678c8bb5-hk2dj"
Nov 03 09:32:15.410700 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.410622    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wt5n\" (UniqueName: \"kubernetes.io/projected/736bfa13-e012-429c-95ff-96b1e83245e6-kube-api-access-8wt5n\") pod \"payload-processing-e2e-autores-85fdd1-64678c8bb5-hk2dj\" (UID: \"736bfa13-e012-429c-95ff-96b1e83245e6\") " pod="openshift-ingress/payload-processing-e2e-autores-85fdd1-64678c8bb5-hk2dj"
Nov 03 09:32:15.413045 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.412499    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/736bfa13-e012-429c-95ff-96b1e83245e6-plugins-config-volume\") pod \"payload-processing-e2e-autores-85fdd1-64678c8bb5-hk2dj\" (UID: \"736bfa13-e012-429c-95ff-96b1e83245e6\") " pod="openshift-ingress/payload-processing-e2e-autores-85fdd1-64678c8bb5-hk2dj"
Nov 03 09:32:15.472845 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.472805    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wt5n\" (UniqueName: \"kubernetes.io/projected/736bfa13-e012-429c-95ff-96b1e83245e6-kube-api-access-8wt5n\") pod \"payload-processing-e2e-autores-85fdd1-64678c8bb5-hk2dj\" (UID: \"736bfa13-e012-429c-95ff-96b1e83245e6\") " pod="openshift-ingress/payload-processing-e2e-autores-85fdd1-64678c8bb5-hk2dj"
Nov 03 09:32:15.538646 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.538426    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-85fdd1-64678c8bb5-hk2dj"
Nov 03 09:32:15.625704 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.625680    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9ed110/auth-test-model-9ed110-kserve-76968d48b9-wlwcm"
Nov 03 09:32:15.719417 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.719390    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dc0565c-837f-432a-bf4e-3b0f611a58fa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dc0565c-837f-432a-bf4e-3b0f611a58fa-dshm\") pod \"9dc0565c-837f-432a-bf4e-3b0f611a58fa\" (UID: \"9dc0565c-837f-432a-bf4e-3b0f611a58fa\") "
Nov 03 09:32:15.719570 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.719492    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dc0565c-837f-432a-bf4e-3b0f611a58fa-home\" (UniqueName: \"kubernetes.io/empty-dir/9dc0565c-837f-432a-bf4e-3b0f611a58fa-home\") pod \"9dc0565c-837f-432a-bf4e-3b0f611a58fa\" (UID: \"9dc0565c-837f-432a-bf4e-3b0f611a58fa\") "
Nov 03 09:32:15.719570 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.719540    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dc0565c-837f-432a-bf4e-3b0f611a58fa-tls-certs\" (UniqueName: \"kubernetes.io/secret/9dc0565c-837f-432a-bf4e-3b0f611a58fa-tls-certs\") pod \"9dc0565c-837f-432a-bf4e-3b0f611a58fa\" (UID: \"9dc0565c-837f-432a-bf4e-3b0f611a58fa\") "
Nov 03 09:32:15.719678 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.719579    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dc0565c-837f-432a-bf4e-3b0f611a58fa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dc0565c-837f-432a-bf4e-3b0f611a58fa-model-cache\") pod \"9dc0565c-837f-432a-bf4e-3b0f611a58fa\" (UID: \"9dc0565c-837f-432a-bf4e-3b0f611a58fa\") "
Nov 03 09:32:15.719678 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.719609    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9dc0565c-837f-432a-bf4e-3b0f611a58fa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dc0565c-837f-432a-bf4e-3b0f611a58fa-tmp-dir\") pod \"9dc0565c-837f-432a-bf4e-3b0f611a58fa\" (UID: \"9dc0565c-837f-432a-bf4e-3b0f611a58fa\") "
Nov 03 09:32:15.719788 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.719768    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dc0565c-837f-432a-bf4e-3b0f611a58fa-kube-api-access-ppzdl\" (UniqueName: \"kubernetes.io/projected/9dc0565c-837f-432a-bf4e-3b0f611a58fa-kube-api-access-ppzdl\") pod \"9dc0565c-837f-432a-bf4e-3b0f611a58fa\" (UID: \"9dc0565c-837f-432a-bf4e-3b0f611a58fa\") "
Nov 03 09:32:15.719788 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.719773    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc0565c-837f-432a-bf4e-3b0f611a58fa-home" pod "9dc0565c-837f-432a-bf4e-3b0f611a58fa" (UID: "9dc0565c-837f-432a-bf4e-3b0f611a58fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:32:15.719973 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.719917    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc0565c-837f-432a-bf4e-3b0f611a58fa-model-cache" pod "9dc0565c-837f-432a-bf4e-3b0f611a58fa" (UID: "9dc0565c-837f-432a-bf4e-3b0f611a58fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:32:15.720035 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.719977    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc0565c-837f-432a-bf4e-3b0f611a58fa-tmp-dir" pod "9dc0565c-837f-432a-bf4e-3b0f611a58fa" (UID: "9dc0565c-837f-432a-bf4e-3b0f611a58fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:32:15.720179 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.720158    4473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dc0565c-837f-432a-bf4e-3b0f611a58fa-home\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:32:15.720283 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.720182    4473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dc0565c-837f-432a-bf4e-3b0f611a58fa-model-cache\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:32:15.720283 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.720197    4473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dc0565c-837f-432a-bf4e-3b0f611a58fa-tmp-dir\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:32:15.722108 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.722085    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dc0565c-837f-432a-bf4e-3b0f611a58fa-kube-api-access-ppzdl" pod "9dc0565c-837f-432a-bf4e-3b0f611a58fa" (UID: "9dc0565c-837f-432a-bf4e-3b0f611a58fa"). InnerVolumeSpecName "kube-api-access-ppzdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:32:15.722260 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.722242    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc0565c-837f-432a-bf4e-3b0f611a58fa-dshm" pod "9dc0565c-837f-432a-bf4e-3b0f611a58fa" (UID: "9dc0565c-837f-432a-bf4e-3b0f611a58fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:32:15.722308 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.722239    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dc0565c-837f-432a-bf4e-3b0f611a58fa-tls-certs" pod "9dc0565c-837f-432a-bf4e-3b0f611a58fa" (UID: "9dc0565c-837f-432a-bf4e-3b0f611a58fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:32:15.776686 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.773681    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-85fdd1-887588c8c-b9dsj"]
Nov 03 09:32:15.822969 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.821295    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppzdl\" (UniqueName: \"kubernetes.io/projected/9dc0565c-837f-432a-bf4e-3b0f611a58fa-kube-api-access-ppzdl\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:32:15.822969 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.821332    4473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dc0565c-837f-432a-bf4e-3b0f611a58fa-dshm\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:32:15.822969 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.821342    4473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dc0565c-837f-432a-bf4e-3b0f611a58fa-tls-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:32:15.876015 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.875980    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="254e50ff-0d1c-48f6-83ca-a88d9effba4d" path="/var/lib/kubelet/pods/254e50ff-0d1c-48f6-83ca-a88d9effba4d/volumes"
Nov 03 09:32:15.969565 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:15.969465    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-85fdd1-64678c8bb5-hk2dj"]
Nov 03 09:32:15.973008 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:32:15.972209    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod736bfa13_e012_429c_95ff_96b1e83245e6.slice/crio-92280dbd08707406601b0f52157866baa30bbe0878c26657f79b4c62bb1bfbd8 WatchSource:0}: Error finding container 92280dbd08707406601b0f52157866baa30bbe0878c26657f79b4c62bb1bfbd8: Status 404 returned error can't find the container with id 92280dbd08707406601b0f52157866baa30bbe0878c26657f79b4c62bb1bfbd8
Nov 03 09:32:16.099399 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:16.099313    4473 generic.go:356] "Generic (PLEG): container finished" podID="9dc0565c-837f-432a-bf4e-3b0f611a58fa" containerID="c5c8564f4150b8febd3bf9e71346d1f198d036cb7296607df6cae603f6e42bd7" exitCode=0
Nov 03 09:32:16.099399 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:16.099361    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9ed110/auth-test-model-9ed110-kserve-76968d48b9-wlwcm"
Nov 03 09:32:16.099618 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:16.099397    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9ed110/auth-test-model-9ed110-kserve-76968d48b9-wlwcm" event={"ID":"9dc0565c-837f-432a-bf4e-3b0f611a58fa","Type":"ContainerDied","Data":"c5c8564f4150b8febd3bf9e71346d1f198d036cb7296607df6cae603f6e42bd7"}
Nov 03 09:32:16.099618 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:16.099451    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9ed110/auth-test-model-9ed110-kserve-76968d48b9-wlwcm" event={"ID":"9dc0565c-837f-432a-bf4e-3b0f611a58fa","Type":"ContainerDied","Data":"db4898dfa588c7c2bf772ad29cf85fe79621f71afe372faea2a6f6b6897779dd"}
Nov 03 09:32:16.099618 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:16.099477    4473 scope.go:122] "RemoveContainer" containerID="c5c8564f4150b8febd3bf9e71346d1f198d036cb7296607df6cae603f6e42bd7"
Nov 03 09:32:16.101553 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:16.101505    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-85fdd1-887588c8c-b9dsj" event={"ID":"e71ebab7-6acf-4f99-bf01-735064ba50e9","Type":"ContainerStarted","Data":"d7ea5140fe00ec33ca30963f75801c04f4630f21dd49ac75105285ee669c3d43"}
Nov 03 09:32:16.101688 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:16.101556    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-85fdd1-887588c8c-b9dsj" event={"ID":"e71ebab7-6acf-4f99-bf01-735064ba50e9","Type":"ContainerStarted","Data":"7fb90ccdfa6ad4ae3b08287eba6691ee7c6137e19a3e2eaba46411c76906167b"}
Nov 03 09:32:16.101983 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:16.101967    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-85fdd1-887588c8c-b9dsj"
Nov 03 09:32:16.103860 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:16.103838    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-85fdd1-64678c8bb5-hk2dj" event={"ID":"736bfa13-e012-429c-95ff-96b1e83245e6","Type":"ContainerStarted","Data":"7181ae0f62025401c54f7877582db95ec31a2b52fe884d2557381805538ec4a3"}
Nov 03 09:32:16.103994 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:16.103880    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-85fdd1-64678c8bb5-hk2dj" event={"ID":"736bfa13-e012-429c-95ff-96b1e83245e6","Type":"ContainerStarted","Data":"92280dbd08707406601b0f52157866baa30bbe0878c26657f79b4c62bb1bfbd8"}
Nov 03 09:32:16.103994 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:16.103911    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-85fdd1-64678c8bb5-hk2dj"
Nov 03 09:32:16.110415 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:16.110341    4473 scope.go:122] "RemoveContainer" containerID="c5c8564f4150b8febd3bf9e71346d1f198d036cb7296607df6cae603f6e42bd7"
Nov 03 09:32:16.110616 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:32:16.110594    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5c8564f4150b8febd3bf9e71346d1f198d036cb7296607df6cae603f6e42bd7\": container with ID starting with c5c8564f4150b8febd3bf9e71346d1f198d036cb7296607df6cae603f6e42bd7 not found: ID does not exist" containerID="c5c8564f4150b8febd3bf9e71346d1f198d036cb7296607df6cae603f6e42bd7"
Nov 03 09:32:16.110680 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:16.110623    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5c8564f4150b8febd3bf9e71346d1f198d036cb7296607df6cae603f6e42bd7"} err="failed to get container status \"c5c8564f4150b8febd3bf9e71346d1f198d036cb7296607df6cae603f6e42bd7\": rpc error: code = NotFound desc = could not find container \"c5c8564f4150b8febd3bf9e71346d1f198d036cb7296607df6cae603f6e42bd7\": container with ID starting with c5c8564f4150b8febd3bf9e71346d1f198d036cb7296607df6cae603f6e42bd7 not found: ID does not exist"
Nov 03 09:32:16.267999 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:16.267962    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9ed110/auth-test-model-9ed110-kserve-76968d48b9-wlwcm"]
Nov 03 09:32:16.275739 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:16.275713    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9ed110/auth-test-model-9ed110-kserve-76968d48b9-wlwcm"]
Nov 03 09:32:16.393180 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:16.393126    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-85fdd1-887588c8c-b9dsj" podStartSLOduration=1.393110363 podStartE2EDuration="1.393110363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:32:16.390451902 +0000 UTC m=+3160.659515357" watchObservedRunningTime="2026-11-03 09:32:16.393110363 +0000 UTC m=+3160.662173796"
Nov 03 09:32:16.516739 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:16.516337    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-85fdd1-64678c8bb5-hk2dj" podStartSLOduration=1.5163148309999999 podStartE2EDuration="1.516314831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:32:16.510394289 +0000 UTC m=+3160.779457756" watchObservedRunningTime="2026-11-03 09:32:16.516314831 +0000 UTC m=+3160.785378282"
Nov 03 09:32:16.937393 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:32:16.937359    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:32:17.870450 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:17.870416    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dc0565c-837f-432a-bf4e-3b0f611a58fa" path="/var/lib/kubelet/pods/9dc0565c-837f-432a-bf4e-3b0f611a58fa/volumes"
Nov 03 09:32:27.116271 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:27.116243    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-85fdd1-887588c8c-b9dsj"
Nov 03 09:32:27.116759 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:27.116319    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-85fdd1-64678c8bb5-hk2dj"
Nov 03 09:32:46.937572 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:32:46.937492    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:32:47.446705 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:47.446665    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:32:51.973267 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:51.973156    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"]
Nov 03 09:32:51.980020 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:51.973677    4473 cpu_manager.go:414] "containerMap: removing container" podUID="9dc0565c-837f-432a-bf4e-3b0f611a58fa" containerName="main"
Nov 03 09:32:51.980020 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:51.973692    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dc0565c-837f-432a-bf4e-3b0f611a58fa" containerName="main"
Nov 03 09:32:51.980020 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:51.973798    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dc0565c-837f-432a-bf4e-3b0f611a58fa" containerName="main"
Nov 03 09:32:51.980020 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:51.978829    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"
Nov 03 09:32:52.025181 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:52.025144    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-istio-podinfo\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h\" (UID: \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"
Nov 03 09:32:52.025318 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:52.025195    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-workload-socket\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h\" (UID: \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"
Nov 03 09:32:52.025373 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:52.025314    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-istio-token\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h\" (UID: \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"
Nov 03 09:32:52.025412 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:52.025385    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxznw\" (UniqueName: \"kubernetes.io/projected/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-kube-api-access-sxznw\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h\" (UID: \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"
Nov 03 09:32:52.025460 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:52.025425    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-workload-certs\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h\" (UID: \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"
Nov 03 09:32:52.025460 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:52.025454    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-istiod-ca-cert\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h\" (UID: \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"
Nov 03 09:32:52.025560 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:52.025538    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-istio-envoy\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h\" (UID: \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"
Nov 03 09:32:52.025604 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:52.025579    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-credential-socket\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h\" (UID: \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"
Nov 03 09:32:52.025642 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:52.025618    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-istio-data\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h\" (UID: \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"
Nov 03 09:32:52.033552 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:52.033528    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-f01a6d-openshift-default-dockercfg-l52zf\""
Nov 03 09:32:52.087534 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:52.087503    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"]
Nov 03 09:32:52.126959 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:52.126918    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxznw\" (UniqueName: \"kubernetes.io/projected/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-kube-api-access-sxznw\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h\" (UID: \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"
Nov 03 09:32:52.127108 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:52.126995    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-workload-certs\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h\" (UID: \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"
Nov 03 09:32:52.127108 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:52.127025    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-istiod-ca-cert\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h\" (UID: \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"
Nov 03 09:32:52.127108 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:52.127072    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-istio-envoy\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h\" (UID: \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"
Nov 03 09:32:52.127252 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:52.127104    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-credential-socket\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h\" (UID: \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"
Nov 03 09:32:52.127252 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:52.127140    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-istio-data\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h\" (UID: \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"
Nov 03 09:32:52.127252 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:52.127229    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-istio-podinfo\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h\" (UID: \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"
Nov 03 09:32:52.127415 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:52.127257    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-workload-socket\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h\" (UID: \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"
Nov 03 09:32:52.127415 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:52.127346    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-istio-token\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h\" (UID: \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"
Nov 03 09:32:52.127525 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:52.127450    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-workload-certs\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h\" (UID: \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"
Nov 03 09:32:52.127591 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:52.127565    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-credential-socket\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h\" (UID: \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"
Nov 03 09:32:52.127640 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:52.127614    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-istio-data\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h\" (UID: \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"
Nov 03 09:32:52.127856 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:52.127837    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-workload-socket\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h\" (UID: \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"
Nov 03 09:32:52.128066 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:52.128045    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-istiod-ca-cert\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h\" (UID: \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"
Nov 03 09:32:52.129841 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:52.129821    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-istio-envoy\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h\" (UID: \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"
Nov 03 09:32:52.130093 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:52.130071    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-istio-podinfo\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h\" (UID: \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"
Nov 03 09:32:52.187503 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:52.187478    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxznw\" (UniqueName: \"kubernetes.io/projected/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-kube-api-access-sxznw\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h\" (UID: \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"
Nov 03 09:32:52.187503 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:52.187497    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-istio-token\") pod \"e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h\" (UID: \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\") " pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"
Nov 03 09:32:52.292721 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:52.292644    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"
Nov 03 09:32:52.724712 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:52.724669    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"]
Nov 03 09:32:52.725783 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:32:52.725758    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d2dfeec_54f4_4fed_a8d8_25f3e50e12fb.slice/crio-a40f75ff813d20f68599f7b5b9f9f2f3ec33e2eeb98a787f280c8f5cedac0086 WatchSource:0}: Error finding container a40f75ff813d20f68599f7b5b9f9f2f3ec33e2eeb98a787f280c8f5cedac0086: Status 404 returned error can't find the container with id a40f75ff813d20f68599f7b5b9f9f2f3ec33e2eeb98a787f280c8f5cedac0086
Nov 03 09:32:52.728379 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:52.728347    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:32:52.728450 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:52.728414    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:32:52.728450 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:52.728442    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:32:53.343839 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:53.343790    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h" event={"ID":"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb","Type":"ContainerStarted","Data":"c1daeafcdf23d4c5900bf0f1dd2c6a7330b7c88d607927cfd307d1c0a7e7f857"}
Nov 03 09:32:53.344395 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:53.343856    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h" event={"ID":"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb","Type":"ContainerStarted","Data":"a40f75ff813d20f68599f7b5b9f9f2f3ec33e2eeb98a787f280c8f5cedac0086"}
Nov 03 09:32:53.465389 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:53.465336    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h" podStartSLOduration=2.465322262 podStartE2EDuration="2.465322262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:32:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:32:53.462361792 +0000 UTC m=+3197.731425245" watchObservedRunningTime="2026-11-03 09:32:53.465322262 +0000 UTC m=+3197.734385696"
Nov 03 09:32:54.293521 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:54.293484    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"
Nov 03 09:32:54.295095 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:54.295074    4473 patch_prober.go:29] interesting pod/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Nov 03 09:32:54.295156 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:54.295121    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h" podUID="3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Nov 03 09:32:55.293319 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:55.293283    4473 patch_prober.go:29] interesting pod/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Nov 03 09:32:55.293718 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:55.293339    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h" podUID="3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Nov 03 09:32:56.300988 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:56.295364    4473 patch_prober.go:29] interesting pod/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Nov 03 09:32:56.300988 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:56.295443    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h" podUID="3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Nov 03 09:32:57.292927 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:57.292889    4473 patch_prober.go:29] interesting pod/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Nov 03 09:32:57.293181 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:57.292967    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h" podUID="3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Nov 03 09:32:58.293845 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:58.293808    4473 patch_prober.go:29] interesting pod/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body=
Nov 03 09:32:58.294241 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:58.293880    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h" podUID="3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused"
Nov 03 09:32:59.297965 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:59.297922    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"
Nov 03 09:32:59.298458 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:59.298435    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"
Nov 03 09:32:59.299107 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:32:59.299088    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"
Nov 03 09:33:07.139139 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:07.139103    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"]
Nov 03 09:33:07.144272 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:07.144251    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"
Nov 03 09:33:07.198802 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:07.198765    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-5adb63-openshift-default-dockercfg-5nfnq\""
Nov 03 09:33:07.224072 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:07.224033    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqnfj\" (UniqueName: \"kubernetes.io/projected/f562628b-a92b-4022-8fe9-4f65cc2f11bb-kube-api-access-qqnfj\") pod \"e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd\" (UID: \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\") " pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"
Nov 03 09:33:07.224525 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:07.224499    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f562628b-a92b-4022-8fe9-4f65cc2f11bb-istio-podinfo\") pod \"e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd\" (UID: \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\") " pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"
Nov 03 09:33:07.224762 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:07.224739    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f562628b-a92b-4022-8fe9-4f65cc2f11bb-istiod-ca-cert\") pod \"e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd\" (UID: \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\") " pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"
Nov 03 09:33:07.233973 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:07.224934    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f562628b-a92b-4022-8fe9-4f65cc2f11bb-istio-token\") pod \"e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd\" (UID: \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\") " pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"
Nov 03 09:33:07.234329 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:07.234303    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f562628b-a92b-4022-8fe9-4f65cc2f11bb-credential-socket\") pod \"e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd\" (UID: \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\") " pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"
Nov 03 09:33:07.234553 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:07.234535    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f562628b-a92b-4022-8fe9-4f65cc2f11bb-workload-socket\") pod \"e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd\" (UID: \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\") " pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"
Nov 03 09:33:07.234747 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:07.234733    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f562628b-a92b-4022-8fe9-4f65cc2f11bb-istio-envoy\") pod \"e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd\" (UID: \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\") " pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"
Nov 03 09:33:07.234891 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:07.234874    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f562628b-a92b-4022-8fe9-4f65cc2f11bb-workload-certs\") pod \"e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd\" (UID: \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\") " pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"
Nov 03 09:33:07.235115 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:07.235095    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f562628b-a92b-4022-8fe9-4f65cc2f11bb-istio-data\") pod \"e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd\" (UID: \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\") " pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"
Nov 03 09:33:07.253670 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:07.253632    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"]
Nov 03 09:33:07.336012 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:07.335981    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f562628b-a92b-4022-8fe9-4f65cc2f11bb-istio-podinfo\") pod \"e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd\" (UID: \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\") " pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"
Nov 03 09:33:07.336207 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:07.336042    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f562628b-a92b-4022-8fe9-4f65cc2f11bb-istiod-ca-cert\") pod \"e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd\" (UID: \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\") " pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"
Nov 03 09:33:07.336207 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:07.336080    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f562628b-a92b-4022-8fe9-4f65cc2f11bb-istio-token\") pod \"e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd\" (UID: \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\") " pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"
Nov 03 09:33:07.336336 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:07.336223    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f562628b-a92b-4022-8fe9-4f65cc2f11bb-credential-socket\") pod \"e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd\" (UID: \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\") " pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"
Nov 03 09:33:07.336336 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:07.336303    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f562628b-a92b-4022-8fe9-4f65cc2f11bb-workload-socket\") pod \"e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd\" (UID: \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\") " pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"
Nov 03 09:33:07.336465 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:07.336365    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f562628b-a92b-4022-8fe9-4f65cc2f11bb-istio-envoy\") pod \"e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd\" (UID: \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\") " pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"
Nov 03 09:33:07.336465 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:07.336394    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f562628b-a92b-4022-8fe9-4f65cc2f11bb-workload-certs\") pod \"e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd\" (UID: \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\") " pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"
Nov 03 09:33:07.336465 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:07.336421    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f562628b-a92b-4022-8fe9-4f65cc2f11bb-istio-data\") pod \"e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd\" (UID: \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\") " pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"
Nov 03 09:33:07.336638 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:07.336519    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqnfj\" (UniqueName: \"kubernetes.io/projected/f562628b-a92b-4022-8fe9-4f65cc2f11bb-kube-api-access-qqnfj\") pod \"e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd\" (UID: \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\") " pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"
Nov 03 09:33:07.336722 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:07.336637    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f562628b-a92b-4022-8fe9-4f65cc2f11bb-credential-socket\") pod \"e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd\" (UID: \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\") " pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"
Nov 03 09:33:07.336816 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:07.336793    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f562628b-a92b-4022-8fe9-4f65cc2f11bb-workload-socket\") pod \"e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd\" (UID: \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\") " pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"
Nov 03 09:33:07.336878 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:07.336859    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f562628b-a92b-4022-8fe9-4f65cc2f11bb-workload-certs\") pod \"e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd\" (UID: \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\") " pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"
Nov 03 09:33:07.337013 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:07.336991    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f562628b-a92b-4022-8fe9-4f65cc2f11bb-istiod-ca-cert\") pod \"e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd\" (UID: \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\") " pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"
Nov 03 09:33:07.337331 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:07.337310    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f562628b-a92b-4022-8fe9-4f65cc2f11bb-istio-data\") pod \"e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd\" (UID: \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\") " pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"
Nov 03 09:33:07.338970 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:07.338923    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f562628b-a92b-4022-8fe9-4f65cc2f11bb-istio-envoy\") pod \"e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd\" (UID: \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\") " pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"
Nov 03 09:33:07.339113 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:07.339059    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f562628b-a92b-4022-8fe9-4f65cc2f11bb-istio-podinfo\") pod \"e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd\" (UID: \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\") " pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"
Nov 03 09:33:07.396453 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:07.396346    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqnfj\" (UniqueName: \"kubernetes.io/projected/f562628b-a92b-4022-8fe9-4f65cc2f11bb-kube-api-access-qqnfj\") pod \"e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd\" (UID: \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\") " pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"
Nov 03 09:33:07.407319 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:07.407289    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f562628b-a92b-4022-8fe9-4f65cc2f11bb-istio-token\") pod \"e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd\" (UID: \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\") " pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"
Nov 03 09:33:07.458700 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:07.458666    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"
Nov 03 09:33:07.889697 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:07.889664    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"]
Nov 03 09:33:07.926900 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:33:07.926853    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf562628b_a92b_4022_8fe9_4f65cc2f11bb.slice/crio-bf260cdbd8856c7fef82a3dbdeb242bafe7850ada1569afa7c025a61882aa747 WatchSource:0}: Error finding container bf260cdbd8856c7fef82a3dbdeb242bafe7850ada1569afa7c025a61882aa747: Status 404 returned error can't find the container with id bf260cdbd8856c7fef82a3dbdeb242bafe7850ada1569afa7c025a61882aa747
Nov 03 09:33:07.929721 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:07.929686    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:33:07.929819 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:07.929756    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:33:07.929819 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:07.929786    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:33:08.456330 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:08.456295    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd" event={"ID":"f562628b-a92b-4022-8fe9-4f65cc2f11bb","Type":"ContainerStarted","Data":"33a4ac9fad5183ab16fc816b3a44ce89b2738c7ba18228888e578e0fc6d3d6b9"}
Nov 03 09:33:08.456742 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:08.456354    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd" event={"ID":"f562628b-a92b-4022-8fe9-4f65cc2f11bb","Type":"ContainerStarted","Data":"bf260cdbd8856c7fef82a3dbdeb242bafe7850ada1569afa7c025a61882aa747"}
Nov 03 09:33:08.459654 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:08.459635    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"
Nov 03 09:33:08.465935 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:08.465914    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"
Nov 03 09:33:08.582838 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:08.582779    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd" podStartSLOduration=1.582765497 podStartE2EDuration="1.582765497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:33:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:33:08.578729012 +0000 UTC m=+3212.847792473" watchObservedRunningTime="2026-11-03 09:33:08.582765497 +0000 UTC m=+3212.851828931"
Nov 03 09:33:09.462193 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:09.462164    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"
Nov 03 09:33:09.463185 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:09.463163    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"
Nov 03 09:33:16.937795 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:33:16.937751    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:33:17.135661 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:17.135627    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5adb63-7767f95b67-5kjcm"]
Nov 03 09:33:17.139891 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:17.139873    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5adb63-7767f95b67-5kjcm"
Nov 03 09:33:17.238478 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:17.238402    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5adb63-7767f95b67-5kjcm"]
Nov 03 09:33:17.239830 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:17.239809    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2375513-57e5-432f-88a5-c61bb7fad533-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-5adb63-7767f95b67-5kjcm\" (UID: \"c2375513-57e5-432f-88a5-c61bb7fad533\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5adb63-7767f95b67-5kjcm"
Nov 03 09:33:17.240016 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:17.239999    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql9ld\" (UniqueName: \"kubernetes.io/projected/c2375513-57e5-432f-88a5-c61bb7fad533-kube-api-access-ql9ld\") pod \"payload-pre-processing-e2e-shared-b-w4-5adb63-7767f95b67-5kjcm\" (UID: \"c2375513-57e5-432f-88a5-c61bb7fad533\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5adb63-7767f95b67-5kjcm"
Nov 03 09:33:17.245881 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:17.245860    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-5adb63\""
Nov 03 09:33:17.246015 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:17.245860    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-5adb63-dockercfg-stzfl\""
Nov 03 09:33:17.347622 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:17.341089    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql9ld\" (UniqueName: \"kubernetes.io/projected/c2375513-57e5-432f-88a5-c61bb7fad533-kube-api-access-ql9ld\") pod \"payload-pre-processing-e2e-shared-b-w4-5adb63-7767f95b67-5kjcm\" (UID: \"c2375513-57e5-432f-88a5-c61bb7fad533\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5adb63-7767f95b67-5kjcm"
Nov 03 09:33:17.347622 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:17.341197    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2375513-57e5-432f-88a5-c61bb7fad533-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-5adb63-7767f95b67-5kjcm\" (UID: \"c2375513-57e5-432f-88a5-c61bb7fad533\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5adb63-7767f95b67-5kjcm"
Nov 03 09:33:17.347622 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:17.342172    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2375513-57e5-432f-88a5-c61bb7fad533-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-5adb63-7767f95b67-5kjcm\" (UID: \"c2375513-57e5-432f-88a5-c61bb7fad533\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5adb63-7767f95b67-5kjcm"
Nov 03 09:33:17.409000 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:17.408965    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql9ld\" (UniqueName: \"kubernetes.io/projected/c2375513-57e5-432f-88a5-c61bb7fad533-kube-api-access-ql9ld\") pod \"payload-pre-processing-e2e-shared-b-w4-5adb63-7767f95b67-5kjcm\" (UID: \"c2375513-57e5-432f-88a5-c61bb7fad533\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5adb63-7767f95b67-5kjcm"
Nov 03 09:33:17.454449 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:17.454422    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5adb63-7767f95b67-5kjcm"
Nov 03 09:33:17.830490 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:17.830452    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5adb63-7767f95b67-5kjcm"]
Nov 03 09:33:17.832203 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:33:17.832175    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2375513_57e5_432f_88a5_c61bb7fad533.slice/crio-546c6e76f91abae5a20e14b95d83368cdd3f0174e4f025b7b3324686a0036b4c WatchSource:0}: Error finding container 546c6e76f91abae5a20e14b95d83368cdd3f0174e4f025b7b3324686a0036b4c: Status 404 returned error can't find the container with id 546c6e76f91abae5a20e14b95d83368cdd3f0174e4f025b7b3324686a0036b4c
Nov 03 09:33:18.531021 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:18.530973    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5adb63-7767f95b67-5kjcm" event={"ID":"c2375513-57e5-432f-88a5-c61bb7fad533","Type":"ContainerStarted","Data":"96b89d3fae8eaeec9992fcc523444eaae79deacc6587b3877ceed568b91a3b5c"}
Nov 03 09:33:18.531442 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:18.531074    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5adb63-7767f95b67-5kjcm"
Nov 03 09:33:18.531442 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:18.531095    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5adb63-7767f95b67-5kjcm" event={"ID":"c2375513-57e5-432f-88a5-c61bb7fad533","Type":"ContainerStarted","Data":"546c6e76f91abae5a20e14b95d83368cdd3f0174e4f025b7b3324686a0036b4c"}
Nov 03 09:33:18.652671 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:18.652607    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5adb63-7767f95b67-5kjcm" podStartSLOduration=1.652588605 podStartE2EDuration="1.652588605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:33:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:33:18.650818617 +0000 UTC m=+3222.919882070" watchObservedRunningTime="2026-11-03 09:33:18.652588605 +0000 UTC m=+3222.921652041"
Nov 03 09:33:29.539827 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:29.539796    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5adb63-7767f95b67-5kjcm"
Nov 03 09:33:46.053037 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:46.052997    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b46dc8f6-t8tvg"]
Nov 03 09:33:46.103221 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:46.103185    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b46dc8f6-t8tvg"]
Nov 03 09:33:46.938220 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:33:46.938184    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:33:47.870157 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:47.870126    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87ddf68f-44eb-473c-aa9d-785716cde3a6" path="/var/lib/kubelet/pods/87ddf68f-44eb-473c-aa9d-785716cde3a6/volumes"
Nov 03 09:33:49.859044 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:49.859004    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f01a6d-55897df747-mvw5h"]
Nov 03 09:33:54.860192 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:54.860144    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f01a6d-55897df747-mvw5h" podUID="c815db09-daba-4d3b-88f2-287d82d328f5" containerName="payload-pre-processing" containerID="cri-o://9652c2bd1d727ee7f5fce7ea1b7de85be57f2c4e147e1ea9c77875bb956ada2a" gracePeriod=25
Nov 03 09:33:55.125762 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:55.125741    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f01a6d-55897df747-mvw5h"
Nov 03 09:33:55.206867 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:55.206839    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c815db09-daba-4d3b-88f2-287d82d328f5-kube-api-access-hdk4g\" (UniqueName: \"kubernetes.io/projected/c815db09-daba-4d3b-88f2-287d82d328f5-kube-api-access-hdk4g\") pod \"c815db09-daba-4d3b-88f2-287d82d328f5\" (UID: \"c815db09-daba-4d3b-88f2-287d82d328f5\") "
Nov 03 09:33:55.207055 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:55.207011    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c815db09-daba-4d3b-88f2-287d82d328f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c815db09-daba-4d3b-88f2-287d82d328f5-plugins-config-volume\") pod \"c815db09-daba-4d3b-88f2-287d82d328f5\" (UID: \"c815db09-daba-4d3b-88f2-287d82d328f5\") "
Nov 03 09:33:55.207334 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:55.207308    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c815db09-daba-4d3b-88f2-287d82d328f5-plugins-config-volume" pod "c815db09-daba-4d3b-88f2-287d82d328f5" (UID: "c815db09-daba-4d3b-88f2-287d82d328f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:33:55.209112 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:55.209092    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c815db09-daba-4d3b-88f2-287d82d328f5-kube-api-access-hdk4g" pod "c815db09-daba-4d3b-88f2-287d82d328f5" (UID: "c815db09-daba-4d3b-88f2-287d82d328f5"). InnerVolumeSpecName "kube-api-access-hdk4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:33:55.308102 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:55.308064    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c815db09-daba-4d3b-88f2-287d82d328f5-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:33:55.308102 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:55.308103    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdk4g\" (UniqueName: \"kubernetes.io/projected/c815db09-daba-4d3b-88f2-287d82d328f5-kube-api-access-hdk4g\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:33:55.789398 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:55.789361    4473 generic.go:356] "Generic (PLEG): container finished" podID="c815db09-daba-4d3b-88f2-287d82d328f5" containerID="9652c2bd1d727ee7f5fce7ea1b7de85be57f2c4e147e1ea9c77875bb956ada2a" exitCode=0
Nov 03 09:33:55.789574 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:55.789410    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f01a6d-55897df747-mvw5h"
Nov 03 09:33:55.789574 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:55.789421    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f01a6d-55897df747-mvw5h" event={"ID":"c815db09-daba-4d3b-88f2-287d82d328f5","Type":"ContainerDied","Data":"9652c2bd1d727ee7f5fce7ea1b7de85be57f2c4e147e1ea9c77875bb956ada2a"}
Nov 03 09:33:55.789574 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:55.789475    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f01a6d-55897df747-mvw5h" event={"ID":"c815db09-daba-4d3b-88f2-287d82d328f5","Type":"ContainerDied","Data":"21f5e8c9c382bdc0c526e102bb6dfdb4069de5a491784f257d89700bdd952944"}
Nov 03 09:33:55.789574 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:55.789504    4473 scope.go:122] "RemoveContainer" containerID="9652c2bd1d727ee7f5fce7ea1b7de85be57f2c4e147e1ea9c77875bb956ada2a"
Nov 03 09:33:55.799439 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:55.799417    4473 scope.go:122] "RemoveContainer" containerID="9652c2bd1d727ee7f5fce7ea1b7de85be57f2c4e147e1ea9c77875bb956ada2a"
Nov 03 09:33:55.799720 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:33:55.799693    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9652c2bd1d727ee7f5fce7ea1b7de85be57f2c4e147e1ea9c77875bb956ada2a\": container with ID starting with 9652c2bd1d727ee7f5fce7ea1b7de85be57f2c4e147e1ea9c77875bb956ada2a not found: ID does not exist" containerID="9652c2bd1d727ee7f5fce7ea1b7de85be57f2c4e147e1ea9c77875bb956ada2a"
Nov 03 09:33:55.799792 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:55.799730    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9652c2bd1d727ee7f5fce7ea1b7de85be57f2c4e147e1ea9c77875bb956ada2a"} err="failed to get container status \"9652c2bd1d727ee7f5fce7ea1b7de85be57f2c4e147e1ea9c77875bb956ada2a\": rpc error: code = NotFound desc = could not find container \"9652c2bd1d727ee7f5fce7ea1b7de85be57f2c4e147e1ea9c77875bb956ada2a\": container with ID starting with 9652c2bd1d727ee7f5fce7ea1b7de85be57f2c4e147e1ea9c77875bb956ada2a not found: ID does not exist"
Nov 03 09:33:55.958783 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:55.958754    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f01a6d-55897df747-mvw5h"]
Nov 03 09:33:56.009507 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:56.009479    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f01a6d-55897df747-mvw5h"]
Nov 03 09:33:57.054068 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:57.054034    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0371fedf-dnjwx"]
Nov 03 09:33:57.104237 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:57.104207    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0371fedf-dnjwx"]
Nov 03 09:33:57.870235 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:57.870205    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b89d43eb-080a-4b2a-91d9-e712bf431043" path="/var/lib/kubelet/pods/b89d43eb-080a-4b2a-91d9-e712bf431043/volumes"
Nov 03 09:33:57.870600 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:57.870588    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c815db09-daba-4d3b-88f2-287d82d328f5" path="/var/lib/kubelet/pods/c815db09-daba-4d3b-88f2-287d82d328f5/volumes"
Nov 03 09:33:58.876513 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:33:58.876474    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:34:09.901410 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:09.901372    4473 generic.go:356] "Generic (PLEG): container finished" podID="fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c" containerID="d560f6b2c2b63d16e7a40fa4597ec897b3ed2f64a9f7bf5193d6ea6701add8ca" exitCode=0
Nov 03 09:34:09.902105 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:09.901443    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st" event={"ID":"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c","Type":"ContainerDied","Data":"d560f6b2c2b63d16e7a40fa4597ec897b3ed2f64a9f7bf5193d6ea6701add8ca"}
Nov 03 09:34:09.902105 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:09.901804    4473 scope.go:122] "RemoveContainer" containerID="d560f6b2c2b63d16e7a40fa4597ec897b3ed2f64a9f7bf5193d6ea6701add8ca"
Nov 03 09:34:09.903423 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:09.903396    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:34:09.903513 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:09.903469    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:34:09.903553 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:09.903502    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:34:10.053272 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:10.053237    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61722479-bp2bm"]
Nov 03 09:34:10.056824 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:10.056788    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61722479-bp2bm"]
Nov 03 09:34:10.910435 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:10.910397    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st" event={"ID":"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c","Type":"ContainerStarted","Data":"e9c469374c25cd3efa15cc803321d826f2ba36ce26b6d2674cd4327e7f50f966"}
Nov 03 09:34:11.141517 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:11.141483    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"
Nov 03 09:34:11.141742 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:11.141720    4473 patch_prober.go:29] interesting pod/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Nov 03 09:34:11.141825 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:11.141774    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st" podUID="fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Nov 03 09:34:11.871023 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:11.870980    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8" path="/var/lib/kubelet/pods/bd1e4f82-6ff0-4b7d-9e23-9dd3779368c8/volumes"
Nov 03 09:34:12.141334 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:12.141241    4473 patch_prober.go:29] interesting pod/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Nov 03 09:34:12.141814 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:12.141324    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st" podUID="fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Nov 03 09:34:13.054447 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:13.054411    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-786f48c9-lvxgh"]
Nov 03 09:34:13.106112 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:13.106083    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-786f48c9-lvxgh"]
Nov 03 09:34:13.141422 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:13.137312    4473 patch_prober.go:29] interesting pod/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Nov 03 09:34:13.141422 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:13.137412    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st" podUID="fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Nov 03 09:34:13.869769 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:13.869735    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74da5b90-e2f4-4a11-82fd-ce5fb88a068c" path="/var/lib/kubelet/pods/74da5b90-e2f4-4a11-82fd-ce5fb88a068c/volumes"
Nov 03 09:34:14.138026 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:14.137288    4473 patch_prober.go:29] interesting pod/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Nov 03 09:34:14.138026 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:14.137351    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st" podUID="fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Nov 03 09:34:15.137971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:15.137277    4473 patch_prober.go:29] interesting pod/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Nov 03 09:34:15.137971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:15.137340    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st" podUID="fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Nov 03 09:34:16.141413 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:16.141376    4473 patch_prober.go:29] interesting pod/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body=
Nov 03 09:34:16.141819 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:16.141444    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st" podUID="fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused"
Nov 03 09:34:16.936634 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:34:16.936560    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:34:17.140895 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:17.140865    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"
Nov 03 09:34:17.141180 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:17.141162    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"
Nov 03 09:34:17.141718 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:17.141699    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"
Nov 03 09:34:21.055238 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:21.055205    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80bbfbc3-hhgbz"]
Nov 03 09:34:21.105831 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:21.105805    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80bbfbc3-hhgbz"]
Nov 03 09:34:21.871152 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:21.871115    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69becb1c-279e-4ef0-b187-ae7f60df3d43" path="/var/lib/kubelet/pods/69becb1c-279e-4ef0-b187-ae7f60df3d43/volumes"
Nov 03 09:34:23.054501 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:23.054462    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bb21c52-cmgl4"]
Nov 03 09:34:23.104728 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:23.104686    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bb21c52-cmgl4"]
Nov 03 09:34:23.869543 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:23.869504    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42779912-fcdc-45e4-bcbf-65e4ae3e3a27" path="/var/lib/kubelet/pods/42779912-fcdc-45e4-bcbf-65e4ae3e3a27/volumes"
Nov 03 09:34:36.494631 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:36.494595    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 09:34:36.496755 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:36.496731    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 09:34:38.268660 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:38.268622    4473 scope.go:122] "RemoveContainer" containerID="c6a843ce30f40bcd9fbcb3c4b2e41a3ddcbe68eb43257dc64bcd6eeadfd4f94a"
Nov 03 09:34:46.937677 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:34:46.937636    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:34:56.675663 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:56.675620    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-739a7f-c5kpq"]
Nov 03 09:34:56.676400 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:56.676378    4473 cpu_manager.go:414] "containerMap: removing container" podUID="c815db09-daba-4d3b-88f2-287d82d328f5" containerName="payload-pre-processing"
Nov 03 09:34:56.676508 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:56.676402    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c815db09-daba-4d3b-88f2-287d82d328f5" containerName="payload-pre-processing"
Nov 03 09:34:56.676586 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:56.676570    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="c815db09-daba-4d3b-88f2-287d82d328f5" containerName="payload-pre-processing"
Nov 03 09:34:56.679939 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:56.679918    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-739a7f-c5kpq"
Nov 03 09:34:56.732471 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:56.732443    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pvjm\" (UniqueName: \"kubernetes.io/projected/c8b2b35b-7e02-4c23-b09d-776c7002b53e-kube-api-access-5pvjm\") pod \"maas-api-revoke-keys-e2e-extmod-a-739a7f-c5kpq\" (UID: \"c8b2b35b-7e02-4c23-b09d-776c7002b53e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-739a7f-c5kpq"
Nov 03 09:34:56.732638 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:56.732481    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b2b35b-7e02-4c23-b09d-776c7002b53e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-739a7f-c5kpq\" (UID: \"c8b2b35b-7e02-4c23-b09d-776c7002b53e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-739a7f-c5kpq"
Nov 03 09:34:56.732638 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:56.732592    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b2b35b-7e02-4c23-b09d-776c7002b53e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-739a7f-c5kpq\" (UID: \"c8b2b35b-7e02-4c23-b09d-776c7002b53e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-739a7f-c5kpq"
Nov 03 09:34:56.740867 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:56.740842    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-739a7f-c5kpq"]
Nov 03 09:34:56.784215 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:56.784182    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:34:56.784370 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:56.784260    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:34:56.784370 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:56.784336    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:34:56.784476 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:56.784379    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:34:56.833823 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:56.833797    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pvjm\" (UniqueName: \"kubernetes.io/projected/c8b2b35b-7e02-4c23-b09d-776c7002b53e-kube-api-access-5pvjm\") pod \"maas-api-revoke-keys-e2e-extmod-a-739a7f-c5kpq\" (UID: \"c8b2b35b-7e02-4c23-b09d-776c7002b53e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-739a7f-c5kpq"
Nov 03 09:34:56.833995 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:56.833829    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b2b35b-7e02-4c23-b09d-776c7002b53e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-739a7f-c5kpq\" (UID: \"c8b2b35b-7e02-4c23-b09d-776c7002b53e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-739a7f-c5kpq"
Nov 03 09:34:56.833995 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:56.833875    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b2b35b-7e02-4c23-b09d-776c7002b53e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-739a7f-c5kpq\" (UID: \"c8b2b35b-7e02-4c23-b09d-776c7002b53e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-739a7f-c5kpq"
Nov 03 09:34:56.834496 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:56.834473    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b2b35b-7e02-4c23-b09d-776c7002b53e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-739a7f-c5kpq\" (UID: \"c8b2b35b-7e02-4c23-b09d-776c7002b53e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-739a7f-c5kpq"
Nov 03 09:34:56.836355 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:56.836338    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b2b35b-7e02-4c23-b09d-776c7002b53e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-739a7f-c5kpq\" (UID: \"c8b2b35b-7e02-4c23-b09d-776c7002b53e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-739a7f-c5kpq"
Nov 03 09:34:56.892979 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:56.892932    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pvjm\" (UniqueName: \"kubernetes.io/projected/c8b2b35b-7e02-4c23-b09d-776c7002b53e-kube-api-access-5pvjm\") pod \"maas-api-revoke-keys-e2e-extmod-a-739a7f-c5kpq\" (UID: \"c8b2b35b-7e02-4c23-b09d-776c7002b53e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-739a7f-c5kpq"
Nov 03 09:34:56.993937 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:56.993872    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-739a7f-c5kpq"
Nov 03 09:34:57.368989 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:57.368956    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-739a7f-c5kpq"]
Nov 03 09:34:57.370059 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:34:57.370032    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b2b35b_7e02_4c23_b09d_776c7002b53e.slice/crio-50397c6842e52184d6fa6eadb3b9ff1e96ca2eb9e4160fbe8e458575957ed6fd WatchSource:0}: Error finding container 50397c6842e52184d6fa6eadb3b9ff1e96ca2eb9e4160fbe8e458575957ed6fd: Status 404 returned error can't find the container with id 50397c6842e52184d6fa6eadb3b9ff1e96ca2eb9e4160fbe8e458575957ed6fd
Nov 03 09:34:58.220638 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:58.220613    4473 generic.go:356] "Generic (PLEG): container finished" podID="c8b2b35b-7e02-4c23-b09d-776c7002b53e" containerID="35178f4984668950c4f9944959121146e89e44c578cf3513024941e6e1cee1c4" exitCode=0
Nov 03 09:34:58.221002 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:58.220674    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-739a7f-c5kpq" event={"ID":"c8b2b35b-7e02-4c23-b09d-776c7002b53e","Type":"ContainerDied","Data":"35178f4984668950c4f9944959121146e89e44c578cf3513024941e6e1cee1c4"}
Nov 03 09:34:58.221002 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:58.220721    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-739a7f-c5kpq" event={"ID":"c8b2b35b-7e02-4c23-b09d-776c7002b53e","Type":"ContainerStarted","Data":"50397c6842e52184d6fa6eadb3b9ff1e96ca2eb9e4160fbe8e458575957ed6fd"}
Nov 03 09:34:59.373092 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:59.373069    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-739a7f-c5kpq"
Nov 03 09:34:59.460744 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:59.460716    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8b2b35b-7e02-4c23-b09d-776c7002b53e-kube-api-access-5pvjm\" (UniqueName: \"kubernetes.io/projected/c8b2b35b-7e02-4c23-b09d-776c7002b53e-kube-api-access-5pvjm\") pod \"c8b2b35b-7e02-4c23-b09d-776c7002b53e\" (UID: \"c8b2b35b-7e02-4c23-b09d-776c7002b53e\") "
Nov 03 09:34:59.460891 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:59.460842    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8b2b35b-7e02-4c23-b09d-776c7002b53e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b2b35b-7e02-4c23-b09d-776c7002b53e-maas-api-openshift-ca\") pod \"c8b2b35b-7e02-4c23-b09d-776c7002b53e\" (UID: \"c8b2b35b-7e02-4c23-b09d-776c7002b53e\") "
Nov 03 09:34:59.460985 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:59.460889    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8b2b35b-7e02-4c23-b09d-776c7002b53e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b2b35b-7e02-4c23-b09d-776c7002b53e-maas-api-xks-ca\") pod \"c8b2b35b-7e02-4c23-b09d-776c7002b53e\" (UID: \"c8b2b35b-7e02-4c23-b09d-776c7002b53e\") "
Nov 03 09:34:59.461196 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:59.461175    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8b2b35b-7e02-4c23-b09d-776c7002b53e-maas-api-openshift-ca" pod "c8b2b35b-7e02-4c23-b09d-776c7002b53e" (UID: "c8b2b35b-7e02-4c23-b09d-776c7002b53e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:34:59.461274 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:59.461261    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b2b35b-7e02-4c23-b09d-776c7002b53e-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:34:59.463054 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:59.463025    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8b2b35b-7e02-4c23-b09d-776c7002b53e-maas-api-xks-ca" pod "c8b2b35b-7e02-4c23-b09d-776c7002b53e" (UID: "c8b2b35b-7e02-4c23-b09d-776c7002b53e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:34:59.463163 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:59.463122    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8b2b35b-7e02-4c23-b09d-776c7002b53e-kube-api-access-5pvjm" pod "c8b2b35b-7e02-4c23-b09d-776c7002b53e" (UID: "c8b2b35b-7e02-4c23-b09d-776c7002b53e"). InnerVolumeSpecName "kube-api-access-5pvjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:34:59.562678 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:59.562605    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b2b35b-7e02-4c23-b09d-776c7002b53e-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:34:59.562678 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:34:59.562630    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pvjm\" (UniqueName: \"kubernetes.io/projected/c8b2b35b-7e02-4c23-b09d-776c7002b53e-kube-api-access-5pvjm\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:35:00.233689 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:00.233601    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-739a7f-c5kpq"
Nov 03 09:35:00.233689 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:00.233631    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-739a7f-c5kpq" event={"ID":"c8b2b35b-7e02-4c23-b09d-776c7002b53e","Type":"ContainerDied","Data":"50397c6842e52184d6fa6eadb3b9ff1e96ca2eb9e4160fbe8e458575957ed6fd"}
Nov 03 09:35:00.233689 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:00.233676    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50397c6842e52184d6fa6eadb3b9ff1e96ca2eb9e4160fbe8e458575957ed6fd"
Nov 03 09:35:10.851906 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:10.851866    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:35:15.025294 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:15.025257    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f01a6d-qw2tq"]
Nov 03 09:35:15.026716 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:15.026689    4473 cpu_manager.go:414] "containerMap: removing container" podUID="c8b2b35b-7e02-4c23-b09d-776c7002b53e" containerName="revoke-keys"
Nov 03 09:35:15.026881 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:15.026867    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8b2b35b-7e02-4c23-b09d-776c7002b53e" containerName="revoke-keys"
Nov 03 09:35:15.027164 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:15.027149    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8b2b35b-7e02-4c23-b09d-776c7002b53e" containerName="revoke-keys"
Nov 03 09:35:15.031035 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:15.031017    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f01a6d-qw2tq"
Nov 03 09:35:15.058878 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:15.058847    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93c0145d-22ce-404c-8491-689495126c00-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f01a6d-qw2tq\" (UID: \"93c0145d-22ce-404c-8491-689495126c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f01a6d-qw2tq"
Nov 03 09:35:15.059096 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:15.059072    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn6jt\" (UniqueName: \"kubernetes.io/projected/93c0145d-22ce-404c-8491-689495126c00-kube-api-access-fn6jt\") pod \"maas-api-revoke-keys-e2e-extmod-a-f01a6d-qw2tq\" (UID: \"93c0145d-22ce-404c-8491-689495126c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f01a6d-qw2tq"
Nov 03 09:35:15.059176 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:15.059128    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93c0145d-22ce-404c-8491-689495126c00-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f01a6d-qw2tq\" (UID: \"93c0145d-22ce-404c-8491-689495126c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f01a6d-qw2tq"
Nov 03 09:35:15.081369 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:15.081336    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f01a6d-qw2tq"]
Nov 03 09:35:15.138327 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:15.138293    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:35:15.138900 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:15.138708    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:35:15.139082 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:15.138771    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:35:15.139201 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:15.138829    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:35:15.160662 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:15.160635    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93c0145d-22ce-404c-8491-689495126c00-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f01a6d-qw2tq\" (UID: \"93c0145d-22ce-404c-8491-689495126c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f01a6d-qw2tq"
Nov 03 09:35:15.160793 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:15.160743    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93c0145d-22ce-404c-8491-689495126c00-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f01a6d-qw2tq\" (UID: \"93c0145d-22ce-404c-8491-689495126c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f01a6d-qw2tq"
Nov 03 09:35:15.160907 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:15.160890    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn6jt\" (UniqueName: \"kubernetes.io/projected/93c0145d-22ce-404c-8491-689495126c00-kube-api-access-fn6jt\") pod \"maas-api-revoke-keys-e2e-extmod-a-f01a6d-qw2tq\" (UID: \"93c0145d-22ce-404c-8491-689495126c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f01a6d-qw2tq"
Nov 03 09:35:15.161412 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:15.161387    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93c0145d-22ce-404c-8491-689495126c00-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f01a6d-qw2tq\" (UID: \"93c0145d-22ce-404c-8491-689495126c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f01a6d-qw2tq"
Nov 03 09:35:15.163344 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:15.163320    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93c0145d-22ce-404c-8491-689495126c00-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f01a6d-qw2tq\" (UID: \"93c0145d-22ce-404c-8491-689495126c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f01a6d-qw2tq"
Nov 03 09:35:15.236970 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:15.233701    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn6jt\" (UniqueName: \"kubernetes.io/projected/93c0145d-22ce-404c-8491-689495126c00-kube-api-access-fn6jt\") pod \"maas-api-revoke-keys-e2e-extmod-a-f01a6d-qw2tq\" (UID: \"93c0145d-22ce-404c-8491-689495126c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f01a6d-qw2tq"
Nov 03 09:35:15.353893 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:15.353800    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f01a6d-qw2tq"
Nov 03 09:35:15.746707 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:15.746671    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f01a6d-qw2tq"]
Nov 03 09:35:16.368669 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:16.368630    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f01a6d-qw2tq" event={"ID":"93c0145d-22ce-404c-8491-689495126c00","Type":"ContainerStarted","Data":"68b9f4b768e9034410a996bbc36ea2919b6c206533dc44f49245ae763b9373dd"}
Nov 03 09:35:16.369192 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:16.368686    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f01a6d-qw2tq" event={"ID":"93c0145d-22ce-404c-8491-689495126c00","Type":"ContainerStarted","Data":"d30f38b96aa1ed4040a90ba616f77986f10b2c40d8fccb165be13d79e9870756"}
Nov 03 09:35:16.487676 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:16.487575    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f01a6d-qw2tq" podStartSLOduration=2.487562164 podStartE2EDuration="2.487562164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:35:16.485637113 +0000 UTC m=+3340.754700569" watchObservedRunningTime="2026-11-03 09:35:16.487562164 +0000 UTC m=+3340.756625618"
Nov 03 09:35:16.963603 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:35:16.963573    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:35:17.376584 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:17.376555    4473 generic.go:356] "Generic (PLEG): container finished" podID="93c0145d-22ce-404c-8491-689495126c00" containerID="68b9f4b768e9034410a996bbc36ea2919b6c206533dc44f49245ae763b9373dd" exitCode=0
Nov 03 09:35:17.376997 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:17.376638    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f01a6d-qw2tq" event={"ID":"93c0145d-22ce-404c-8491-689495126c00","Type":"ContainerDied","Data":"68b9f4b768e9034410a996bbc36ea2919b6c206533dc44f49245ae763b9373dd"}
Nov 03 09:35:18.572274 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:18.572252    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f01a6d-qw2tq"
Nov 03 09:35:18.604119 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:18.604097    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93c0145d-22ce-404c-8491-689495126c00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93c0145d-22ce-404c-8491-689495126c00-maas-api-xks-ca\") pod \"93c0145d-22ce-404c-8491-689495126c00\" (UID: \"93c0145d-22ce-404c-8491-689495126c00\") "
Nov 03 09:35:18.604263 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:18.604220    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93c0145d-22ce-404c-8491-689495126c00-kube-api-access-fn6jt\" (UniqueName: \"kubernetes.io/projected/93c0145d-22ce-404c-8491-689495126c00-kube-api-access-fn6jt\") pod \"93c0145d-22ce-404c-8491-689495126c00\" (UID: \"93c0145d-22ce-404c-8491-689495126c00\") "
Nov 03 09:35:18.604349 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:18.604336    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93c0145d-22ce-404c-8491-689495126c00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93c0145d-22ce-404c-8491-689495126c00-maas-api-openshift-ca\") pod \"93c0145d-22ce-404c-8491-689495126c00\" (UID: \"93c0145d-22ce-404c-8491-689495126c00\") "
Nov 03 09:35:18.604670 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:18.604650    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93c0145d-22ce-404c-8491-689495126c00-maas-api-openshift-ca" pod "93c0145d-22ce-404c-8491-689495126c00" (UID: "93c0145d-22ce-404c-8491-689495126c00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:35:18.606297 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:18.606272    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93c0145d-22ce-404c-8491-689495126c00-maas-api-xks-ca" pod "93c0145d-22ce-404c-8491-689495126c00" (UID: "93c0145d-22ce-404c-8491-689495126c00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:35:18.606414 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:18.606369    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93c0145d-22ce-404c-8491-689495126c00-kube-api-access-fn6jt" pod "93c0145d-22ce-404c-8491-689495126c00" (UID: "93c0145d-22ce-404c-8491-689495126c00"). InnerVolumeSpecName "kube-api-access-fn6jt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:35:18.705845 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:18.705759    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93c0145d-22ce-404c-8491-689495126c00-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:35:18.705845 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:18.705803    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn6jt\" (UniqueName: \"kubernetes.io/projected/93c0145d-22ce-404c-8491-689495126c00-kube-api-access-fn6jt\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:35:18.705845 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:18.705823    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93c0145d-22ce-404c-8491-689495126c00-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:35:19.390530 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:19.390500    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f01a6d-qw2tq"
Nov 03 09:35:19.390706 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:19.390499    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f01a6d-qw2tq" event={"ID":"93c0145d-22ce-404c-8491-689495126c00","Type":"ContainerDied","Data":"d30f38b96aa1ed4040a90ba616f77986f10b2c40d8fccb165be13d79e9870756"}
Nov 03 09:35:19.390706 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:19.390630    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d30f38b96aa1ed4040a90ba616f77986f10b2c40d8fccb165be13d79e9870756"
Nov 03 09:35:22.412710 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:22.412665    4473 generic.go:356] "Generic (PLEG): container finished" podID="f562628b-a92b-4022-8fe9-4f65cc2f11bb" containerID="33a4ac9fad5183ab16fc816b3a44ce89b2738c7ba18228888e578e0fc6d3d6b9" exitCode=0
Nov 03 09:35:22.413263 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:22.412751    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd" event={"ID":"f562628b-a92b-4022-8fe9-4f65cc2f11bb","Type":"ContainerDied","Data":"33a4ac9fad5183ab16fc816b3a44ce89b2738c7ba18228888e578e0fc6d3d6b9"}
Nov 03 09:35:22.413263 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:22.413226    4473 scope.go:122] "RemoveContainer" containerID="33a4ac9fad5183ab16fc816b3a44ce89b2738c7ba18228888e578e0fc6d3d6b9"
Nov 03 09:35:22.415269 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:22.415237    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:35:22.415353 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:22.415328    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:35:22.415409 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:22.415372    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:35:23.442969 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:23.433025    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd" event={"ID":"f562628b-a92b-4022-8fe9-4f65cc2f11bb","Type":"ContainerStarted","Data":"4893a04622108de79d791a5e6a1394ffb2bc0151a613a4bdca0c02ca63712696"}
Nov 03 09:35:23.459523 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:23.459496    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"
Nov 03 09:35:23.460809 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:23.460781    4473 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Nov 03 09:35:23.460930 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:23.460832    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd" podUID="f562628b-a92b-4022-8fe9-4f65cc2f11bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Nov 03 09:35:24.460073 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:24.460043    4473 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Nov 03 09:35:24.460472 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:24.460101    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd" podUID="f562628b-a92b-4022-8fe9-4f65cc2f11bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Nov 03 09:35:24.462254 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:24.462228    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"
Nov 03 09:35:25.459076 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:25.459044    4473 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Nov 03 09:35:25.459270 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:25.459096    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd" podUID="f562628b-a92b-4022-8fe9-4f65cc2f11bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Nov 03 09:35:26.459918 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:26.459884    4473 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Nov 03 09:35:26.460350 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:26.459968    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd" podUID="f562628b-a92b-4022-8fe9-4f65cc2f11bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Nov 03 09:35:27.459901 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:27.459866    4473 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Nov 03 09:35:27.460129 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:27.459915    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd" podUID="f562628b-a92b-4022-8fe9-4f65cc2f11bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Nov 03 09:35:28.459519 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:28.459476    4473 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Nov 03 09:35:28.459719 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:28.459544    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd" podUID="f562628b-a92b-4022-8fe9-4f65cc2f11bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Nov 03 09:35:29.371672 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:29.371634    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-85fdd1-64678c8bb5-hk2dj"]
Nov 03 09:35:29.464184 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:29.464145    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"
Nov 03 09:35:29.465113 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:29.465091    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"
Nov 03 09:35:29.485717 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:29.485691    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-85fdd1-887588c8c-b9dsj"]
Nov 03 09:35:34.372161 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:34.372120    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-85fdd1-64678c8bb5-hk2dj" podUID="736bfa13-e012-429c-95ff-96b1e83245e6" containerName="payload-processing" containerID="cri-o://7181ae0f62025401c54f7877582db95ec31a2b52fe884d2557381805538ec4a3" gracePeriod=25
Nov 03 09:35:34.486858 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:34.486793    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-85fdd1-887588c8c-b9dsj" podUID="e71ebab7-6acf-4f99-bf01-735064ba50e9" containerName="payload-pre-processing" containerID="cri-o://d7ea5140fe00ec33ca30963f75801c04f4630f21dd49ac75105285ee669c3d43" gracePeriod=25
Nov 03 09:35:34.519626 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:34.519595    4473 generic.go:356] "Generic (PLEG): container finished" podID="736bfa13-e012-429c-95ff-96b1e83245e6" containerID="7181ae0f62025401c54f7877582db95ec31a2b52fe884d2557381805538ec4a3" exitCode=0
Nov 03 09:35:34.519781 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:34.519666    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-85fdd1-64678c8bb5-hk2dj" event={"ID":"736bfa13-e012-429c-95ff-96b1e83245e6","Type":"ContainerDied","Data":"7181ae0f62025401c54f7877582db95ec31a2b52fe884d2557381805538ec4a3"}
Nov 03 09:35:34.647162 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:34.647135    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-85fdd1-64678c8bb5-hk2dj"
Nov 03 09:35:34.726616 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:34.726587    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/736bfa13-e012-429c-95ff-96b1e83245e6-kube-api-access-8wt5n\" (UniqueName: \"kubernetes.io/projected/736bfa13-e012-429c-95ff-96b1e83245e6-kube-api-access-8wt5n\") pod \"736bfa13-e012-429c-95ff-96b1e83245e6\" (UID: \"736bfa13-e012-429c-95ff-96b1e83245e6\") "
Nov 03 09:35:34.726809 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:34.726791    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/736bfa13-e012-429c-95ff-96b1e83245e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/736bfa13-e012-429c-95ff-96b1e83245e6-plugins-config-volume\") pod \"736bfa13-e012-429c-95ff-96b1e83245e6\" (UID: \"736bfa13-e012-429c-95ff-96b1e83245e6\") "
Nov 03 09:35:34.727207 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:34.727179    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/736bfa13-e012-429c-95ff-96b1e83245e6-plugins-config-volume" pod "736bfa13-e012-429c-95ff-96b1e83245e6" (UID: "736bfa13-e012-429c-95ff-96b1e83245e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:35:34.727464 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:34.727444    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/736bfa13-e012-429c-95ff-96b1e83245e6-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:35:34.729519 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:34.729492    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/736bfa13-e012-429c-95ff-96b1e83245e6-kube-api-access-8wt5n" pod "736bfa13-e012-429c-95ff-96b1e83245e6" (UID: "736bfa13-e012-429c-95ff-96b1e83245e6"). InnerVolumeSpecName "kube-api-access-8wt5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:35:34.749432 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:34.749415    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-85fdd1-887588c8c-b9dsj"
Nov 03 09:35:34.828622 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:34.828589    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e71ebab7-6acf-4f99-bf01-735064ba50e9-kube-api-access-lxks7\" (UniqueName: \"kubernetes.io/projected/e71ebab7-6acf-4f99-bf01-735064ba50e9-kube-api-access-lxks7\") pod \"e71ebab7-6acf-4f99-bf01-735064ba50e9\" (UID: \"e71ebab7-6acf-4f99-bf01-735064ba50e9\") "
Nov 03 09:35:34.828806 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:34.828740    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e71ebab7-6acf-4f99-bf01-735064ba50e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e71ebab7-6acf-4f99-bf01-735064ba50e9-plugins-config-volume\") pod \"e71ebab7-6acf-4f99-bf01-735064ba50e9\" (UID: \"e71ebab7-6acf-4f99-bf01-735064ba50e9\") "
Nov 03 09:35:34.829057 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:34.829043    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wt5n\" (UniqueName: \"kubernetes.io/projected/736bfa13-e012-429c-95ff-96b1e83245e6-kube-api-access-8wt5n\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:35:34.829126 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:34.829106    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e71ebab7-6acf-4f99-bf01-735064ba50e9-plugins-config-volume" pod "e71ebab7-6acf-4f99-bf01-735064ba50e9" (UID: "e71ebab7-6acf-4f99-bf01-735064ba50e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:35:34.830818 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:34.830798    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e71ebab7-6acf-4f99-bf01-735064ba50e9-kube-api-access-lxks7" pod "e71ebab7-6acf-4f99-bf01-735064ba50e9" (UID: "e71ebab7-6acf-4f99-bf01-735064ba50e9"). InnerVolumeSpecName "kube-api-access-lxks7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:35:34.929692 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:34.929626    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e71ebab7-6acf-4f99-bf01-735064ba50e9-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:35:34.929692 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:34.929650    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxks7\" (UniqueName: \"kubernetes.io/projected/e71ebab7-6acf-4f99-bf01-735064ba50e9-kube-api-access-lxks7\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:35:35.527443 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:35.527415    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-85fdd1-64678c8bb5-hk2dj"
Nov 03 09:35:35.527899 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:35.527443    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-85fdd1-64678c8bb5-hk2dj" event={"ID":"736bfa13-e012-429c-95ff-96b1e83245e6","Type":"ContainerDied","Data":"92280dbd08707406601b0f52157866baa30bbe0878c26657f79b4c62bb1bfbd8"}
Nov 03 09:35:35.527899 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:35.527496    4473 scope.go:122] "RemoveContainer" containerID="7181ae0f62025401c54f7877582db95ec31a2b52fe884d2557381805538ec4a3"
Nov 03 09:35:35.529133 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:35.529108    4473 generic.go:356] "Generic (PLEG): container finished" podID="e71ebab7-6acf-4f99-bf01-735064ba50e9" containerID="d7ea5140fe00ec33ca30963f75801c04f4630f21dd49ac75105285ee669c3d43" exitCode=0
Nov 03 09:35:35.529237 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:35.529155    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-85fdd1-887588c8c-b9dsj"
Nov 03 09:35:35.529237 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:35.529184    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-85fdd1-887588c8c-b9dsj" event={"ID":"e71ebab7-6acf-4f99-bf01-735064ba50e9","Type":"ContainerDied","Data":"d7ea5140fe00ec33ca30963f75801c04f4630f21dd49ac75105285ee669c3d43"}
Nov 03 09:35:35.529237 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:35.529229    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-85fdd1-887588c8c-b9dsj" event={"ID":"e71ebab7-6acf-4f99-bf01-735064ba50e9","Type":"ContainerDied","Data":"7fb90ccdfa6ad4ae3b08287eba6691ee7c6137e19a3e2eaba46411c76906167b"}
Nov 03 09:35:35.539680 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:35.539662    4473 scope.go:122] "RemoveContainer" containerID="d7ea5140fe00ec33ca30963f75801c04f4630f21dd49ac75105285ee669c3d43"
Nov 03 09:35:35.551797 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:35.551778    4473 scope.go:122] "RemoveContainer" containerID="d7ea5140fe00ec33ca30963f75801c04f4630f21dd49ac75105285ee669c3d43"
Nov 03 09:35:35.552120 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:35:35.552101    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7ea5140fe00ec33ca30963f75801c04f4630f21dd49ac75105285ee669c3d43\": container with ID starting with d7ea5140fe00ec33ca30963f75801c04f4630f21dd49ac75105285ee669c3d43 not found: ID does not exist" containerID="d7ea5140fe00ec33ca30963f75801c04f4630f21dd49ac75105285ee669c3d43"
Nov 03 09:35:35.552181 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:35.552130    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7ea5140fe00ec33ca30963f75801c04f4630f21dd49ac75105285ee669c3d43"} err="failed to get container status \"d7ea5140fe00ec33ca30963f75801c04f4630f21dd49ac75105285ee669c3d43\": rpc error: code = NotFound desc = could not find container \"d7ea5140fe00ec33ca30963f75801c04f4630f21dd49ac75105285ee669c3d43\": container with ID starting with d7ea5140fe00ec33ca30963f75801c04f4630f21dd49ac75105285ee669c3d43 not found: ID does not exist"
Nov 03 09:35:35.696261 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:35.696189    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-85fdd1-887588c8c-b9dsj"]
Nov 03 09:35:35.746644 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:35.746611    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-85fdd1-887588c8c-b9dsj"]
Nov 03 09:35:35.814592 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:35.814551    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-85fdd1-64678c8bb5-hk2dj"]
Nov 03 09:35:35.817733 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:35.817707    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-85fdd1-64678c8bb5-hk2dj"]
Nov 03 09:35:35.870933 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:35.870902    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="736bfa13-e012-429c-95ff-96b1e83245e6" path="/var/lib/kubelet/pods/736bfa13-e012-429c-95ff-96b1e83245e6/volumes"
Nov 03 09:35:35.871400 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:35.871379    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e71ebab7-6acf-4f99-bf01-735064ba50e9" path="/var/lib/kubelet/pods/e71ebab7-6acf-4f99-bf01-735064ba50e9/volumes"
Nov 03 09:35:38.305399 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:38.305364    4473 scope.go:122] "RemoveContainer" containerID="769aaf5198e318896766b017e84819e3927b9a9b3faf1b34783dff4380611e86"
Nov 03 09:35:38.315174 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:38.315152    4473 scope.go:122] "RemoveContainer" containerID="78451d534563c58ccec8763542b38417f9d480c261030512472794901f1c83d5"
Nov 03 09:35:38.324100 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:38.324079    4473 scope.go:122] "RemoveContainer" containerID="436b9af9de4373f07a3e26bce7fef15a12df1d39203b71609f2e35d3746d46da"
Nov 03 09:35:38.332464 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:38.332447    4473 scope.go:122] "RemoveContainer" containerID="902a9af4b01f6b369ade740aa06bfc85ae7dda0fb964c67f067e4fe4f6044aa6"
Nov 03 09:35:38.340559 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:38.340538    4473 scope.go:122] "RemoveContainer" containerID="bdfd1224747d7cc1bb17bacca2aeeb6fa443b7e8875a9768ded19826a8e8348b"
Nov 03 09:35:38.348819 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:38.348800    4473 scope.go:122] "RemoveContainer" containerID="9ecce1799df0b8f5d78f041f66de83b161e180bec6fbbdc376ec85dfaf88dec2"
Nov 03 09:35:43.360101 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:35:43.360072    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:35:46.937373 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:35:46.937287    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:36:08.052365 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:08.052334    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b47771b-52jsh"]
Nov 03 09:36:08.102665 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:08.102640    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b47771b-52jsh"]
Nov 03 09:36:09.869914 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:09.869878    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e55c656f-62c0-46fd-a33b-362d89080a95" path="/var/lib/kubelet/pods/e55c656f-62c0-46fd-a33b-362d89080a95/volumes"
Nov 03 09:36:16.937059 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:36:16.937020    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:36:19.060973 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:19.060926    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-486ceaf4-xwz8x"]
Nov 03 09:36:19.112051 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:19.112021    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-486ceaf4-xwz8x"]
Nov 03 09:36:19.870127 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:19.870095    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9772119f-bb0f-44fa-8277-211ff4c2d429" path="/var/lib/kubelet/pods/9772119f-bb0f-44fa-8277-211ff4c2d429/volumes"
Nov 03 09:36:20.790970 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.790929    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"]
Nov 03 09:36:20.791445 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.791432    4473 cpu_manager.go:414] "containerMap: removing container" podUID="736bfa13-e012-429c-95ff-96b1e83245e6" containerName="payload-processing"
Nov 03 09:36:20.791527 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.791446    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="736bfa13-e012-429c-95ff-96b1e83245e6" containerName="payload-processing"
Nov 03 09:36:20.791527 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.791458    4473 cpu_manager.go:414] "containerMap: removing container" podUID="93c0145d-22ce-404c-8491-689495126c00" containerName="revoke-keys"
Nov 03 09:36:20.791527 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.791463    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93c0145d-22ce-404c-8491-689495126c00" containerName="revoke-keys"
Nov 03 09:36:20.791527 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.791494    4473 cpu_manager.go:414] "containerMap: removing container" podUID="e71ebab7-6acf-4f99-bf01-735064ba50e9" containerName="payload-pre-processing"
Nov 03 09:36:20.791527 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.791500    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e71ebab7-6acf-4f99-bf01-735064ba50e9" containerName="payload-pre-processing"
Nov 03 09:36:20.791740 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.791589    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="736bfa13-e012-429c-95ff-96b1e83245e6" containerName="payload-processing"
Nov 03 09:36:20.791740 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.791602    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="93c0145d-22ce-404c-8491-689495126c00" containerName="revoke-keys"
Nov 03 09:36:20.791740 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.791621    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="e71ebab7-6acf-4f99-bf01-735064ba50e9" containerName="payload-pre-processing"
Nov 03 09:36:20.795164 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.795147    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"
Nov 03 09:36:20.826211 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.826182    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gvlpl\" (UID: \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"
Nov 03 09:36:20.826356 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.826291    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gvlpl\" (UID: \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"
Nov 03 09:36:20.826356 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.826341    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gvlpl\" (UID: \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"
Nov 03 09:36:20.826527 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.826437    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gvlpl\" (UID: \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"
Nov 03 09:36:20.826527 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.826468    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gvlpl\" (UID: \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"
Nov 03 09:36:20.826650 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.826558    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gvlpl\" (UID: \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"
Nov 03 09:36:20.826650 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.826587    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jn5n\" (UniqueName: \"kubernetes.io/projected/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-kube-api-access-2jn5n\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gvlpl\" (UID: \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"
Nov 03 09:36:20.826650 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.826613    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gvlpl\" (UID: \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"
Nov 03 09:36:20.826752 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.826711    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gvlpl\" (UID: \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"
Nov 03 09:36:20.850248 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.850228    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-pqxg4\""
Nov 03 09:36:20.903355 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.903329    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"]
Nov 03 09:36:20.927446 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.927415    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gvlpl\" (UID: \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"
Nov 03 09:36:20.927621 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.927471    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jn5n\" (UniqueName: \"kubernetes.io/projected/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-kube-api-access-2jn5n\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gvlpl\" (UID: \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"
Nov 03 09:36:20.927621 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.927521    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gvlpl\" (UID: \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"
Nov 03 09:36:20.927621 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.927575    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gvlpl\" (UID: \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"
Nov 03 09:36:20.927821 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.927634    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gvlpl\" (UID: \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"
Nov 03 09:36:20.927821 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.927706    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gvlpl\" (UID: \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"
Nov 03 09:36:20.927821 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.927753    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gvlpl\" (UID: \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"
Nov 03 09:36:20.927821 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.927797    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gvlpl\" (UID: \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"
Nov 03 09:36:20.928071 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.927825    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gvlpl\" (UID: \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"
Nov 03 09:36:20.928071 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.927918    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gvlpl\" (UID: \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"
Nov 03 09:36:20.928071 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.928021    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gvlpl\" (UID: \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"
Nov 03 09:36:20.928249 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.928197    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gvlpl\" (UID: \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"
Nov 03 09:36:20.928573 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.928531    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gvlpl\" (UID: \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"
Nov 03 09:36:20.928675 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.928595    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gvlpl\" (UID: \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"
Nov 03 09:36:20.930766 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.930739    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gvlpl\" (UID: \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"
Nov 03 09:36:20.931024 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.931001    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gvlpl\" (UID: \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"
Nov 03 09:36:20.988591 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.988560    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gvlpl\" (UID: \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"
Nov 03 09:36:20.989316 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:20.989298    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jn5n\" (UniqueName: \"kubernetes.io/projected/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-kube-api-access-2jn5n\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gvlpl\" (UID: \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"
Nov 03 09:36:21.108916 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:21.108842    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"
Nov 03 09:36:21.538245 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:21.538217    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"]
Nov 03 09:36:21.539389 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:36:21.539362    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b80f43c_c1d9_4f07_84eb_7aa145b3d03c.slice/crio-b42a30d197fb2ac0f478ef20622a36fc0c7a2a694b7c8709f206889aff8c9e06 WatchSource:0}: Error finding container b42a30d197fb2ac0f478ef20622a36fc0c7a2a694b7c8709f206889aff8c9e06: Status 404 returned error can't find the container with id b42a30d197fb2ac0f478ef20622a36fc0c7a2a694b7c8709f206889aff8c9e06
Nov 03 09:36:21.541905 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:21.541875    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:36:21.542006 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:21.541971    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:36:21.542058 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:21.542006    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:36:21.795423 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:21.795345    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl" event={"ID":"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c","Type":"ContainerStarted","Data":"7e0e27efcf2d0388eb1dc4d9013c9b575739d7fd33a1f1e391294b6ab2775852"}
Nov 03 09:36:21.795423 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:21.795394    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl" event={"ID":"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c","Type":"ContainerStarted","Data":"b42a30d197fb2ac0f478ef20622a36fc0c7a2a694b7c8709f206889aff8c9e06"}
Nov 03 09:36:21.919628 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:21.919567    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl" podStartSLOduration=1.9195481380000001 podStartE2EDuration="1.919548138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:36:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:36:21.913889932 +0000 UTC m=+3406.182953367" watchObservedRunningTime="2026-11-03 09:36:21.919548138 +0000 UTC m=+3406.188611572"
Nov 03 09:36:22.109818 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:22.109727    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"
Nov 03 09:36:22.111004 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:22.110978    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 03 09:36:22.111135 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:22.111029    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl" podUID="1b80f43c-c1d9-4f07-84eb-7aa145b3d03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 03 09:36:23.113348 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:23.113303    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 03 09:36:23.113751 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:23.113380    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl" podUID="1b80f43c-c1d9-4f07-84eb-7aa145b3d03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 03 09:36:24.109316 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:24.109272    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 03 09:36:24.109527 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:24.109345    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl" podUID="1b80f43c-c1d9-4f07-84eb-7aa145b3d03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 03 09:36:25.109621 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:25.109586    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 03 09:36:25.110057 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:25.109636    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl" podUID="1b80f43c-c1d9-4f07-84eb-7aa145b3d03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 03 09:36:26.109141 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:26.109108    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 03 09:36:26.109328 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:26.109169    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl" podUID="1b80f43c-c1d9-4f07-84eb-7aa145b3d03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 03 09:36:27.109334 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:27.109297    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 03 09:36:27.109723 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:27.109354    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl" podUID="1b80f43c-c1d9-4f07-84eb-7aa145b3d03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 03 09:36:28.110239 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:28.110197    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 03 09:36:28.110785 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:28.110272    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl" podUID="1b80f43c-c1d9-4f07-84eb-7aa145b3d03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 03 09:36:29.109444 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:29.109408    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body=
Nov 03 09:36:29.109756 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:29.109728    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl" podUID="1b80f43c-c1d9-4f07-84eb-7aa145b3d03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused"
Nov 03 09:36:30.319692 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:30.319656    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-739a7f-c5kpq"]
Nov 03 09:36:30.371787 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:30.371747    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-739a7f-c5kpq"]
Nov 03 09:36:30.578278 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:30.578198    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"
Nov 03 09:36:30.580241 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:30.580218    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"
Nov 03 09:36:30.581436 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:30.581418    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"
Nov 03 09:36:31.870439 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:31.870407    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8b2b35b-7e02-4c23-b09d-776c7002b53e" path="/var/lib/kubelet/pods/c8b2b35b-7e02-4c23-b09d-776c7002b53e/volumes"
Nov 03 09:36:43.382701 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:43.382672    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f01a6d-qw2tq"]
Nov 03 09:36:43.432740 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:43.432713    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f01a6d-qw2tq"]
Nov 03 09:36:43.594188 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:43.594154    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"]
Nov 03 09:36:43.594422 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:43.594389    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h" podUID="3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb" containerName="istio-proxy" containerID="cri-o://c1daeafcdf23d4c5900bf0f1dd2c6a7330b7c88d607927cfd307d1c0a7e7f857" gracePeriod=30
Nov 03 09:36:43.870423 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:43.870394    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93c0145d-22ce-404c-8491-689495126c00" path="/var/lib/kubelet/pods/93c0145d-22ce-404c-8491-689495126c00/volumes"
Nov 03 09:36:44.301012 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:44.300898    4473 patch_prober.go:29] interesting pod/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 09:36:44.301012 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:44.300995    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h" podUID="3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 09:36:46.952500 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:36:46.952410    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:36:47.746473 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:47.746442    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:36:48.900126 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:48.900102    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"
Nov 03 09:36:48.990961 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:48.990911    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-credential-socket\") pod \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\" (UID: \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\") "
Nov 03 09:36:48.991148 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:48.990972    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-kube-api-access-sxznw\" (UniqueName: \"kubernetes.io/projected/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-kube-api-access-sxznw\") pod \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\" (UID: \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\") "
Nov 03 09:36:48.991148 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:48.991012    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-workload-socket\") pod \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\" (UID: \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\") "
Nov 03 09:36:48.991268 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:48.991155    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-istio-token\" (UniqueName: \"kubernetes.io/projected/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-istio-token\") pod \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\" (UID: \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\") "
Nov 03 09:36:48.991268 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:48.991187    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-credential-socket" pod "3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb" (UID: "3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:36:48.991268 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:48.991223    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-workload-certs\") pod \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\" (UID: \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\") "
Nov 03 09:36:48.991268 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:48.991244    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-istiod-ca-cert\") pod \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\" (UID: \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\") "
Nov 03 09:36:48.991491 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:48.991288    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-istio-envoy\") pod \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\" (UID: \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\") "
Nov 03 09:36:48.991491 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:48.991329    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-istio-data\") pod \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\" (UID: \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\") "
Nov 03 09:36:48.991491 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:48.991350    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-workload-socket" pod "3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb" (UID: "3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:36:48.991491 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:48.991457    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-istio-podinfo\") pod \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\" (UID: \"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb\") "
Nov 03 09:36:48.991752 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:48.991493    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-workload-certs" pod "3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb" (UID: "3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:36:48.991752 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:48.991623    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-istio-data" pod "3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb" (UID: "3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:36:48.992027 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:48.992002    4473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-workload-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:36:48.992177 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:48.992031    4473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-istio-data\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:36:48.992177 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:48.992047    4473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-credential-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:36:48.992177 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:48.992061    4473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-workload-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:36:48.992177 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:48.992071    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-istiod-ca-cert" pod "3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb" (UID: "3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:36:48.993776 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:48.993756    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-istio-podinfo" pod "3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb" (UID: "3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 09:36:48.993981 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:48.993935    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-istio-token" pod "3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb" (UID: "3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:36:48.994073 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:48.994061    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-kube-api-access-sxznw" pod "3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb" (UID: "3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb"). InnerVolumeSpecName "kube-api-access-sxznw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:36:48.994148 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:48.994127    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-istio-envoy" pod "3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb" (UID: "3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:36:49.026045 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:49.026008    4473 generic.go:356] "Generic (PLEG): container finished" podID="3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb" containerID="c1daeafcdf23d4c5900bf0f1dd2c6a7330b7c88d607927cfd307d1c0a7e7f857" exitCode=0
Nov 03 09:36:49.026214 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:49.026060    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"
Nov 03 09:36:49.026214 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:49.026070    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h" event={"ID":"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb","Type":"ContainerDied","Data":"c1daeafcdf23d4c5900bf0f1dd2c6a7330b7c88d607927cfd307d1c0a7e7f857"}
Nov 03 09:36:49.026214 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:49.026125    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h" event={"ID":"3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb","Type":"ContainerDied","Data":"a40f75ff813d20f68599f7b5b9f9f2f3ec33e2eeb98a787f280c8f5cedac0086"}
Nov 03 09:36:49.026214 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:49.026154    4473 scope.go:122] "RemoveContainer" containerID="c1daeafcdf23d4c5900bf0f1dd2c6a7330b7c88d607927cfd307d1c0a7e7f857"
Nov 03 09:36:49.037876 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:49.037768    4473 scope.go:122] "RemoveContainer" containerID="c1daeafcdf23d4c5900bf0f1dd2c6a7330b7c88d607927cfd307d1c0a7e7f857"
Nov 03 09:36:49.038337 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:36:49.038103    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1daeafcdf23d4c5900bf0f1dd2c6a7330b7c88d607927cfd307d1c0a7e7f857\": container with ID starting with c1daeafcdf23d4c5900bf0f1dd2c6a7330b7c88d607927cfd307d1c0a7e7f857 not found: ID does not exist" containerID="c1daeafcdf23d4c5900bf0f1dd2c6a7330b7c88d607927cfd307d1c0a7e7f857"
Nov 03 09:36:49.038337 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:49.038135    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1daeafcdf23d4c5900bf0f1dd2c6a7330b7c88d607927cfd307d1c0a7e7f857"} err="failed to get container status \"c1daeafcdf23d4c5900bf0f1dd2c6a7330b7c88d607927cfd307d1c0a7e7f857\": rpc error: code = NotFound desc = could not find container \"c1daeafcdf23d4c5900bf0f1dd2c6a7330b7c88d607927cfd307d1c0a7e7f857\": container with ID starting with c1daeafcdf23d4c5900bf0f1dd2c6a7330b7c88d607927cfd307d1c0a7e7f857 not found: ID does not exist"
Nov 03 09:36:49.093352 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:49.093317    4473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-istiod-ca-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:36:49.093352 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:49.093346    4473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-istio-envoy\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:36:49.093352 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:49.093358    4473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-istio-podinfo\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:36:49.093581 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:49.093366    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxznw\" (UniqueName: \"kubernetes.io/projected/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-kube-api-access-sxznw\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:36:49.093581 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:49.093376    4473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb-istio-token\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:36:49.235549 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:49.235470    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"]
Nov 03 09:36:49.242510 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:49.242484    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-f01a6d-openshift-default-5ff955f6cc-h4n4h"]
Nov 03 09:36:49.873141 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:49.873095    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb" path="/var/lib/kubelet/pods/3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb/volumes"
Nov 03 09:36:52.320339 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:52.320304    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"]
Nov 03 09:36:52.320864 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:52.320512    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st" podUID="fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c" containerName="istio-proxy" containerID="cri-o://e9c469374c25cd3efa15cc803321d826f2ba36ce26b6d2674cd4327e7f50f966" gracePeriod=30
Nov 03 09:36:57.594680 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:57.594622    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"
Nov 03 09:36:57.686665 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:57.686634    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-credential-socket\") pod \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\" (UID: \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\") "
Nov 03 09:36:57.686839 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:57.686714    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-istiod-ca-cert\") pod \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\" (UID: \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\") "
Nov 03 09:36:57.686839 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:57.686748    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-istio-data\") pod \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\" (UID: \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\") "
Nov 03 09:36:57.686976 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:57.686838    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-workload-certs\") pod \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\" (UID: \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\") "
Nov 03 09:36:57.686976 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:57.686884    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-kube-api-access-q6w6h\" (UniqueName: \"kubernetes.io/projected/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-kube-api-access-q6w6h\") pod \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\" (UID: \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\") "
Nov 03 09:36:57.686976 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:57.686909    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-credential-socket" pod "fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c" (UID: "fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:36:57.686976 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:57.686917    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-istio-envoy\") pod \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\" (UID: \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\") "
Nov 03 09:36:57.687208 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:57.687025    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-workload-socket\") pod \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\" (UID: \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\") "
Nov 03 09:36:57.687208 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:57.687097    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-istio-token\" (UniqueName: \"kubernetes.io/projected/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-istio-token\") pod \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\" (UID: \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\") "
Nov 03 09:36:57.687208 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:57.687126    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-istiod-ca-cert" pod "fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c" (UID: "fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:36:57.687208 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:57.687139    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-istio-podinfo\") pod \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\" (UID: \"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c\") "
Nov 03 09:36:57.687208 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:57.687159    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-istio-data" pod "fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c" (UID: "fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:36:57.687497 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:57.687352    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-workload-socket" pod "fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c" (UID: "fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:36:57.687651 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:57.687631    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-workload-certs" pod "fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c" (UID: "fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:36:57.687962 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:57.687805    4473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-workload-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:36:57.687962 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:57.687899    4473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-credential-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:36:57.687962 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:57.687915    4473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-istiod-ca-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:36:57.687962 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:57.687930    4473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-istio-data\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:36:57.689329 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:57.689303    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-istio-envoy" pod "fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c" (UID: "fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:36:57.689424 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:57.689370    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-istio-token" pod "fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c" (UID: "fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:36:57.689634 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:57.689607    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-istio-podinfo" pod "fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c" (UID: "fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 09:36:57.689745 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:57.689693    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-kube-api-access-q6w6h" pod "fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c" (UID: "fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c"). InnerVolumeSpecName "kube-api-access-q6w6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:36:57.789055 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:57.789023    4473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-workload-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:36:57.789055 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:57.789053    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6w6h\" (UniqueName: \"kubernetes.io/projected/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-kube-api-access-q6w6h\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:36:57.789055 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:57.789064    4473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-istio-envoy\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:36:57.789300 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:57.789073    4473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-istio-token\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:36:57.789300 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:57.789081    4473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c-istio-podinfo\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:36:58.091630 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:58.091548    4473 generic.go:356] "Generic (PLEG): container finished" podID="fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c" containerID="e9c469374c25cd3efa15cc803321d826f2ba36ce26b6d2674cd4327e7f50f966" exitCode=0
Nov 03 09:36:58.091630 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:58.091605    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"
Nov 03 09:36:58.091850 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:58.091628    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st" event={"ID":"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c","Type":"ContainerDied","Data":"e9c469374c25cd3efa15cc803321d826f2ba36ce26b6d2674cd4327e7f50f966"}
Nov 03 09:36:58.091850 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:58.091686    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st" event={"ID":"fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c","Type":"ContainerDied","Data":"1f05277119fdf2560e86d3d15be519a87c6b2da770377f5383cef16b8a4cd2bb"}
Nov 03 09:36:58.091850 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:58.091708    4473 scope.go:122] "RemoveContainer" containerID="e9c469374c25cd3efa15cc803321d826f2ba36ce26b6d2674cd4327e7f50f966"
Nov 03 09:36:58.101917 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:58.101898    4473 scope.go:122] "RemoveContainer" containerID="d560f6b2c2b63d16e7a40fa4597ec897b3ed2f64a9f7bf5193d6ea6701add8ca"
Nov 03 09:36:58.111009 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:58.110992    4473 scope.go:122] "RemoveContainer" containerID="e9c469374c25cd3efa15cc803321d826f2ba36ce26b6d2674cd4327e7f50f966"
Nov 03 09:36:58.111270 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:36:58.111248    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9c469374c25cd3efa15cc803321d826f2ba36ce26b6d2674cd4327e7f50f966\": container with ID starting with e9c469374c25cd3efa15cc803321d826f2ba36ce26b6d2674cd4327e7f50f966 not found: ID does not exist" containerID="e9c469374c25cd3efa15cc803321d826f2ba36ce26b6d2674cd4327e7f50f966"
Nov 03 09:36:58.111346 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:58.111278    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9c469374c25cd3efa15cc803321d826f2ba36ce26b6d2674cd4327e7f50f966"} err="failed to get container status \"e9c469374c25cd3efa15cc803321d826f2ba36ce26b6d2674cd4327e7f50f966\": rpc error: code = NotFound desc = could not find container \"e9c469374c25cd3efa15cc803321d826f2ba36ce26b6d2674cd4327e7f50f966\": container with ID starting with e9c469374c25cd3efa15cc803321d826f2ba36ce26b6d2674cd4327e7f50f966 not found: ID does not exist"
Nov 03 09:36:58.111346 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:58.111299    4473 scope.go:122] "RemoveContainer" containerID="d560f6b2c2b63d16e7a40fa4597ec897b3ed2f64a9f7bf5193d6ea6701add8ca"
Nov 03 09:36:58.111558 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:36:58.111536    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d560f6b2c2b63d16e7a40fa4597ec897b3ed2f64a9f7bf5193d6ea6701add8ca\": container with ID starting with d560f6b2c2b63d16e7a40fa4597ec897b3ed2f64a9f7bf5193d6ea6701add8ca not found: ID does not exist" containerID="d560f6b2c2b63d16e7a40fa4597ec897b3ed2f64a9f7bf5193d6ea6701add8ca"
Nov 03 09:36:58.111605 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:58.111564    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d560f6b2c2b63d16e7a40fa4597ec897b3ed2f64a9f7bf5193d6ea6701add8ca"} err="failed to get container status \"d560f6b2c2b63d16e7a40fa4597ec897b3ed2f64a9f7bf5193d6ea6701add8ca\": rpc error: code = NotFound desc = could not find container \"d560f6b2c2b63d16e7a40fa4597ec897b3ed2f64a9f7bf5193d6ea6701add8ca\": container with ID starting with d560f6b2c2b63d16e7a40fa4597ec897b3ed2f64a9f7bf5193d6ea6701add8ca not found: ID does not exist"
Nov 03 09:36:58.261657 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:58.261584    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"]
Nov 03 09:36:58.266264 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:58.266237    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-85fdd1-openshift-default-586f5945f5-g94st"]
Nov 03 09:36:59.870419 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:36:59.870388    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c" path="/var/lib/kubelet/pods/fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c/volumes"
Nov 03 09:37:01.304928 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:01.304891    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8af98dc3-2hsmf"]
Nov 03 09:37:01.305406 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:01.305392    4473 cpu_manager.go:414] "containerMap: removing container" podUID="fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c" containerName="istio-proxy"
Nov 03 09:37:01.305465 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:01.305407    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c" containerName="istio-proxy"
Nov 03 09:37:01.305465 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:01.305435    4473 cpu_manager.go:414] "containerMap: removing container" podUID="3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb" containerName="istio-proxy"
Nov 03 09:37:01.305465 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:01.305440    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb" containerName="istio-proxy"
Nov 03 09:37:01.305465 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:01.305451    4473 cpu_manager.go:414] "containerMap: removing container" podUID="fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c" containerName="istio-proxy"
Nov 03 09:37:01.305465 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:01.305456    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c" containerName="istio-proxy"
Nov 03 09:37:01.305799 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:01.305522    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d2dfeec-54f4-4fed-a8d8-25f3e50e12fb" containerName="istio-proxy"
Nov 03 09:37:01.305799 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:01.305530    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c" containerName="istio-proxy"
Nov 03 09:37:01.310747 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:01.310721    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af98dc3-2hsmf"
Nov 03 09:37:01.366255 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:01.366230    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8af98dc3-2hsmf"]
Nov 03 09:37:01.413124 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:01.413099    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:37:01.413298 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:01.413132    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:37:01.414327 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:01.414309    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:37:01.414459 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:01.414443    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:37:01.426739 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:01.426716    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06434827-ae83-450d-8afa-104d54b147b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8af98dc3-2hsmf\" (UID: \"06434827-ae83-450d-8afa-104d54b147b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af98dc3-2hsmf"
Nov 03 09:37:01.426931 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:01.426911    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06434827-ae83-450d-8afa-104d54b147b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8af98dc3-2hsmf\" (UID: \"06434827-ae83-450d-8afa-104d54b147b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af98dc3-2hsmf"
Nov 03 09:37:01.427026 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:01.427000    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdhx6\" (UniqueName: \"kubernetes.io/projected/06434827-ae83-450d-8afa-104d54b147b6-kube-api-access-mdhx6\") pod \"maas-api-revoke-sub-8af98dc3-2hsmf\" (UID: \"06434827-ae83-450d-8afa-104d54b147b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af98dc3-2hsmf"
Nov 03 09:37:01.527664 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:01.527627    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06434827-ae83-450d-8afa-104d54b147b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8af98dc3-2hsmf\" (UID: \"06434827-ae83-450d-8afa-104d54b147b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af98dc3-2hsmf"
Nov 03 09:37:01.527845 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:01.527679    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdhx6\" (UniqueName: \"kubernetes.io/projected/06434827-ae83-450d-8afa-104d54b147b6-kube-api-access-mdhx6\") pod \"maas-api-revoke-sub-8af98dc3-2hsmf\" (UID: \"06434827-ae83-450d-8afa-104d54b147b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af98dc3-2hsmf"
Nov 03 09:37:01.527845 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:01.527749    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06434827-ae83-450d-8afa-104d54b147b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8af98dc3-2hsmf\" (UID: \"06434827-ae83-450d-8afa-104d54b147b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af98dc3-2hsmf"
Nov 03 09:37:01.528403 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:01.528376    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06434827-ae83-450d-8afa-104d54b147b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8af98dc3-2hsmf\" (UID: \"06434827-ae83-450d-8afa-104d54b147b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af98dc3-2hsmf"
Nov 03 09:37:01.530176 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:01.530157    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06434827-ae83-450d-8afa-104d54b147b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8af98dc3-2hsmf\" (UID: \"06434827-ae83-450d-8afa-104d54b147b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af98dc3-2hsmf"
Nov 03 09:37:01.587654 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:01.587593    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdhx6\" (UniqueName: \"kubernetes.io/projected/06434827-ae83-450d-8afa-104d54b147b6-kube-api-access-mdhx6\") pod \"maas-api-revoke-sub-8af98dc3-2hsmf\" (UID: \"06434827-ae83-450d-8afa-104d54b147b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af98dc3-2hsmf"
Nov 03 09:37:01.624374 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:01.624346    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af98dc3-2hsmf"
Nov 03 09:37:02.002172 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:02.002138    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8af98dc3-2hsmf"]
Nov 03 09:37:02.003928 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:37:02.003897    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06434827_ae83_450d_8afa_104d54b147b6.slice/crio-aa0ff8deea2ddf08ecd8ef295244219634bddd953f5de209c4cfea1d42a5136d WatchSource:0}: Error finding container aa0ff8deea2ddf08ecd8ef295244219634bddd953f5de209c4cfea1d42a5136d: Status 404 returned error can't find the container with id aa0ff8deea2ddf08ecd8ef295244219634bddd953f5de209c4cfea1d42a5136d
Nov 03 09:37:02.119480 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:02.119436    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af98dc3-2hsmf" event={"ID":"06434827-ae83-450d-8afa-104d54b147b6","Type":"ContainerStarted","Data":"aa0ff8deea2ddf08ecd8ef295244219634bddd953f5de209c4cfea1d42a5136d"}
Nov 03 09:37:03.127605 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:03.127575    4473 generic.go:356] "Generic (PLEG): container finished" podID="06434827-ae83-450d-8afa-104d54b147b6" containerID="0ed7e42671dccdf2c188e8d1e8b79003081d6dd5f888d23754b5bd0efa5953b0" exitCode=0
Nov 03 09:37:03.128027 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:03.127677    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af98dc3-2hsmf" event={"ID":"06434827-ae83-450d-8afa-104d54b147b6","Type":"ContainerDied","Data":"0ed7e42671dccdf2c188e8d1e8b79003081d6dd5f888d23754b5bd0efa5953b0"}
Nov 03 09:37:03.535394 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:03.535359    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-373116-759d64f4b7-r4dmq"]
Nov 03 09:37:03.535983 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:03.535969    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbf6f0a2-6f2d-40ce-801c-ff3830b0c75c" containerName="istio-proxy"
Nov 03 09:37:03.539285 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:03.539265    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-373116-759d64f4b7-r4dmq"
Nov 03 09:37:03.597081 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:03.597046    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-373116\""
Nov 03 09:37:03.598920 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:03.598884    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-373116-759d64f4b7-r4dmq"]
Nov 03 09:37:03.662260 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:03.662227    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-373116-dockercfg-tr8xk\""
Nov 03 09:37:03.672028 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:03.663891    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f5b134-d645-43dc-a388-10e05dc52168-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-373116-759d64f4b7-r4dmq\" (UID: \"d6f5b134-d645-43dc-a388-10e05dc52168\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-373116-759d64f4b7-r4dmq"
Nov 03 09:37:03.673651 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:03.673620    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7l62\" (UniqueName: \"kubernetes.io/projected/d6f5b134-d645-43dc-a388-10e05dc52168-kube-api-access-q7l62\") pod \"payload-pre-processing-e2e-extmod-a-373116-759d64f4b7-r4dmq\" (UID: \"d6f5b134-d645-43dc-a388-10e05dc52168\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-373116-759d64f4b7-r4dmq"
Nov 03 09:37:03.676985 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:03.676940    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-373116-5645958765-96jxp"]
Nov 03 09:37:03.681734 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:03.681714    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-373116-5645958765-96jxp"
Nov 03 09:37:03.728333 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:03.728305    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-373116-5645958765-96jxp"]
Nov 03 09:37:03.775249 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:03.775214    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7l62\" (UniqueName: \"kubernetes.io/projected/d6f5b134-d645-43dc-a388-10e05dc52168-kube-api-access-q7l62\") pod \"payload-pre-processing-e2e-extmod-a-373116-759d64f4b7-r4dmq\" (UID: \"d6f5b134-d645-43dc-a388-10e05dc52168\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-373116-759d64f4b7-r4dmq"
Nov 03 09:37:03.775435 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:03.775277    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bxws\" (UniqueName: \"kubernetes.io/projected/2ff5c62e-0eaa-45d9-86db-8636677353fc-kube-api-access-6bxws\") pod \"payload-processing-e2e-extmod-a-373116-5645958765-96jxp\" (UID: \"2ff5c62e-0eaa-45d9-86db-8636677353fc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-373116-5645958765-96jxp"
Nov 03 09:37:03.775435 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:03.775299    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f5b134-d645-43dc-a388-10e05dc52168-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-373116-759d64f4b7-r4dmq\" (UID: \"d6f5b134-d645-43dc-a388-10e05dc52168\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-373116-759d64f4b7-r4dmq"
Nov 03 09:37:03.775593 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:03.775451    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ff5c62e-0eaa-45d9-86db-8636677353fc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-373116-5645958765-96jxp\" (UID: \"2ff5c62e-0eaa-45d9-86db-8636677353fc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-373116-5645958765-96jxp"
Nov 03 09:37:03.776074 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:03.776058    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f5b134-d645-43dc-a388-10e05dc52168-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-373116-759d64f4b7-r4dmq\" (UID: \"d6f5b134-d645-43dc-a388-10e05dc52168\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-373116-759d64f4b7-r4dmq"
Nov 03 09:37:03.834400 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:03.834327    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7l62\" (UniqueName: \"kubernetes.io/projected/d6f5b134-d645-43dc-a388-10e05dc52168-kube-api-access-q7l62\") pod \"payload-pre-processing-e2e-extmod-a-373116-759d64f4b7-r4dmq\" (UID: \"d6f5b134-d645-43dc-a388-10e05dc52168\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-373116-759d64f4b7-r4dmq"
Nov 03 09:37:03.852867 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:03.852843    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-373116-759d64f4b7-r4dmq"
Nov 03 09:37:03.876468 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:03.876440    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bxws\" (UniqueName: \"kubernetes.io/projected/2ff5c62e-0eaa-45d9-86db-8636677353fc-kube-api-access-6bxws\") pod \"payload-processing-e2e-extmod-a-373116-5645958765-96jxp\" (UID: \"2ff5c62e-0eaa-45d9-86db-8636677353fc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-373116-5645958765-96jxp"
Nov 03 09:37:03.876590 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:03.876569    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ff5c62e-0eaa-45d9-86db-8636677353fc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-373116-5645958765-96jxp\" (UID: \"2ff5c62e-0eaa-45d9-86db-8636677353fc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-373116-5645958765-96jxp"
Nov 03 09:37:03.877295 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:03.877277    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ff5c62e-0eaa-45d9-86db-8636677353fc-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-373116-5645958765-96jxp\" (UID: \"2ff5c62e-0eaa-45d9-86db-8636677353fc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-373116-5645958765-96jxp"
Nov 03 09:37:03.936090 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:03.936061    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bxws\" (UniqueName: \"kubernetes.io/projected/2ff5c62e-0eaa-45d9-86db-8636677353fc-kube-api-access-6bxws\") pod \"payload-processing-e2e-extmod-a-373116-5645958765-96jxp\" (UID: \"2ff5c62e-0eaa-45d9-86db-8636677353fc\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-373116-5645958765-96jxp"
Nov 03 09:37:03.997991 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:03.997939    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-373116-5645958765-96jxp"
Nov 03 09:37:04.233236 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:04.233205    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-373116-759d64f4b7-r4dmq"]
Nov 03 09:37:04.234592 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:37:04.234559    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6f5b134_d645_43dc_a388_10e05dc52168.slice/crio-4ce10b2c0b25351c521d410e6053fa79f79fac30dbe42468d80f8c66d1e5f759 WatchSource:0}: Error finding container 4ce10b2c0b25351c521d410e6053fa79f79fac30dbe42468d80f8c66d1e5f759: Status 404 returned error can't find the container with id 4ce10b2c0b25351c521d410e6053fa79f79fac30dbe42468d80f8c66d1e5f759
Nov 03 09:37:04.278372 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:04.278352    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af98dc3-2hsmf"
Nov 03 09:37:04.376081 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:04.376056    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-373116-5645958765-96jxp"]
Nov 03 09:37:04.377380 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:37:04.377357    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff5c62e_0eaa_45d9_86db_8636677353fc.slice/crio-4bccfcd96198eedbc15045c925b19d081939560c6ce198d9ffc25b8cd9dbe823 WatchSource:0}: Error finding container 4bccfcd96198eedbc15045c925b19d081939560c6ce198d9ffc25b8cd9dbe823: Status 404 returned error can't find the container with id 4bccfcd96198eedbc15045c925b19d081939560c6ce198d9ffc25b8cd9dbe823
Nov 03 09:37:04.381903 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:04.381884    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06434827-ae83-450d-8afa-104d54b147b6-kube-api-access-mdhx6\" (UniqueName: \"kubernetes.io/projected/06434827-ae83-450d-8afa-104d54b147b6-kube-api-access-mdhx6\") pod \"06434827-ae83-450d-8afa-104d54b147b6\" (UID: \"06434827-ae83-450d-8afa-104d54b147b6\") "
Nov 03 09:37:04.382068 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:04.382053    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06434827-ae83-450d-8afa-104d54b147b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06434827-ae83-450d-8afa-104d54b147b6-maas-api-openshift-ca\") pod \"06434827-ae83-450d-8afa-104d54b147b6\" (UID: \"06434827-ae83-450d-8afa-104d54b147b6\") "
Nov 03 09:37:04.382145 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:04.382097    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06434827-ae83-450d-8afa-104d54b147b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06434827-ae83-450d-8afa-104d54b147b6-maas-api-xks-ca\") pod \"06434827-ae83-450d-8afa-104d54b147b6\" (UID: \"06434827-ae83-450d-8afa-104d54b147b6\") "
Nov 03 09:37:04.382399 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:04.382379    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06434827-ae83-450d-8afa-104d54b147b6-maas-api-openshift-ca" pod "06434827-ae83-450d-8afa-104d54b147b6" (UID: "06434827-ae83-450d-8afa-104d54b147b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:37:04.384126 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:04.384083    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06434827-ae83-450d-8afa-104d54b147b6-kube-api-access-mdhx6" pod "06434827-ae83-450d-8afa-104d54b147b6" (UID: "06434827-ae83-450d-8afa-104d54b147b6"). InnerVolumeSpecName "kube-api-access-mdhx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:37:04.384300 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:04.384275    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06434827-ae83-450d-8afa-104d54b147b6-maas-api-xks-ca" pod "06434827-ae83-450d-8afa-104d54b147b6" (UID: "06434827-ae83-450d-8afa-104d54b147b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:37:04.483242 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:04.483218    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06434827-ae83-450d-8afa-104d54b147b6-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:37:04.483242 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:04.483243    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06434827-ae83-450d-8afa-104d54b147b6-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:37:04.483407 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:04.483271    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdhx6\" (UniqueName: \"kubernetes.io/projected/06434827-ae83-450d-8afa-104d54b147b6-kube-api-access-mdhx6\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:37:05.142513 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:05.142483    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-373116-5645958765-96jxp" event={"ID":"2ff5c62e-0eaa-45d9-86db-8636677353fc","Type":"ContainerStarted","Data":"3eea2681da7998aa52762905937334aa65bbb172f7c5bf4aa7ee5392e39521dc"}
Nov 03 09:37:05.142715 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:05.142533    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-373116-5645958765-96jxp" event={"ID":"2ff5c62e-0eaa-45d9-86db-8636677353fc","Type":"ContainerStarted","Data":"4bccfcd96198eedbc15045c925b19d081939560c6ce198d9ffc25b8cd9dbe823"}
Nov 03 09:37:05.142715 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:05.142593    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-373116-5645958765-96jxp"
Nov 03 09:37:05.144198 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:05.144171    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-373116-759d64f4b7-r4dmq" event={"ID":"d6f5b134-d645-43dc-a388-10e05dc52168","Type":"ContainerStarted","Data":"67abebd98d7a9bc3dd69f963825de232ddc90b1f84f8691bbc6ce4945924269b"}
Nov 03 09:37:05.144333 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:05.144225    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-373116-759d64f4b7-r4dmq"
Nov 03 09:37:05.144333 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:05.144252    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-373116-759d64f4b7-r4dmq" event={"ID":"d6f5b134-d645-43dc-a388-10e05dc52168","Type":"ContainerStarted","Data":"4ce10b2c0b25351c521d410e6053fa79f79fac30dbe42468d80f8c66d1e5f759"}
Nov 03 09:37:05.145471 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:05.145453    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af98dc3-2hsmf"
Nov 03 09:37:05.145567 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:05.145466    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af98dc3-2hsmf" event={"ID":"06434827-ae83-450d-8afa-104d54b147b6","Type":"ContainerDied","Data":"aa0ff8deea2ddf08ecd8ef295244219634bddd953f5de209c4cfea1d42a5136d"}
Nov 03 09:37:05.145567 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:05.145505    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa0ff8deea2ddf08ecd8ef295244219634bddd953f5de209c4cfea1d42a5136d"
Nov 03 09:37:05.265296 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:05.265248    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-373116-5645958765-96jxp" podStartSLOduration=2.265234769 podStartE2EDuration="2.265234769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:37:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:37:05.261340924 +0000 UTC m=+3449.530404378" watchObservedRunningTime="2026-11-03 09:37:05.265234769 +0000 UTC m=+3449.534298181"
Nov 03 09:37:05.420202 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:05.420100    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-373116-759d64f4b7-r4dmq" podStartSLOduration=2.4200815 podStartE2EDuration="2.4200815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:37:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:37:05.397208426 +0000 UTC m=+3449.666271861" watchObservedRunningTime="2026-11-03 09:37:05.4200815 +0000 UTC m=+3449.689144936"
Nov 03 09:37:06.153239 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:06.153205    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"]
Nov 03 09:37:06.153764 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:06.153748    4473 cpu_manager.go:414] "containerMap: removing container" podUID="06434827-ae83-450d-8afa-104d54b147b6" containerName="revoke-keys"
Nov 03 09:37:06.153855 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:06.153768    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06434827-ae83-450d-8afa-104d54b147b6" containerName="revoke-keys"
Nov 03 09:37:06.153917 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:06.153894    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="06434827-ae83-450d-8afa-104d54b147b6" containerName="revoke-keys"
Nov 03 09:37:06.157402 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:06.157377    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"
Nov 03 09:37:06.223371 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:06.223344    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm8xh\" (UniqueName: \"kubernetes.io/projected/7e4c0879-52a1-4038-aedd-3810b70a76d6-kube-api-access-cm8xh\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wchmh\" (UID: \"7e4c0879-52a1-4038-aedd-3810b70a76d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"
Nov 03 09:37:06.223507 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:06.223381    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0879-52a1-4038-aedd-3810b70a76d6-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wchmh\" (UID: \"7e4c0879-52a1-4038-aedd-3810b70a76d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"
Nov 03 09:37:06.223507 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:06.223411    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0879-52a1-4038-aedd-3810b70a76d6-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wchmh\" (UID: \"7e4c0879-52a1-4038-aedd-3810b70a76d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"
Nov 03 09:37:06.223609 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:06.223560    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0879-52a1-4038-aedd-3810b70a76d6-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wchmh\" (UID: \"7e4c0879-52a1-4038-aedd-3810b70a76d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"
Nov 03 09:37:06.223655 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:06.223628    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e4c0879-52a1-4038-aedd-3810b70a76d6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wchmh\" (UID: \"7e4c0879-52a1-4038-aedd-3810b70a76d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"
Nov 03 09:37:06.223655 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:06.223646    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e4c0879-52a1-4038-aedd-3810b70a76d6-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wchmh\" (UID: \"7e4c0879-52a1-4038-aedd-3810b70a76d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"
Nov 03 09:37:06.223724 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:06.223665    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e4c0879-52a1-4038-aedd-3810b70a76d6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wchmh\" (UID: \"7e4c0879-52a1-4038-aedd-3810b70a76d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"
Nov 03 09:37:06.223892 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:06.223875    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0879-52a1-4038-aedd-3810b70a76d6-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wchmh\" (UID: \"7e4c0879-52a1-4038-aedd-3810b70a76d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"
Nov 03 09:37:06.223991 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:06.223975    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0879-52a1-4038-aedd-3810b70a76d6-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wchmh\" (UID: \"7e4c0879-52a1-4038-aedd-3810b70a76d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"
Nov 03 09:37:06.325428 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:06.325392    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0879-52a1-4038-aedd-3810b70a76d6-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wchmh\" (UID: \"7e4c0879-52a1-4038-aedd-3810b70a76d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"
Nov 03 09:37:06.325883 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:06.325438    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0879-52a1-4038-aedd-3810b70a76d6-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wchmh\" (UID: \"7e4c0879-52a1-4038-aedd-3810b70a76d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"
Nov 03 09:37:06.325883 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:06.325526    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e4c0879-52a1-4038-aedd-3810b70a76d6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wchmh\" (UID: \"7e4c0879-52a1-4038-aedd-3810b70a76d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"
Nov 03 09:37:06.325883 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:06.325556    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e4c0879-52a1-4038-aedd-3810b70a76d6-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wchmh\" (UID: \"7e4c0879-52a1-4038-aedd-3810b70a76d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"
Nov 03 09:37:06.325883 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:06.325588    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e4c0879-52a1-4038-aedd-3810b70a76d6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wchmh\" (UID: \"7e4c0879-52a1-4038-aedd-3810b70a76d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"
Nov 03 09:37:06.325883 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:06.325648    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0879-52a1-4038-aedd-3810b70a76d6-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wchmh\" (UID: \"7e4c0879-52a1-4038-aedd-3810b70a76d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"
Nov 03 09:37:06.325883 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:06.325709    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0879-52a1-4038-aedd-3810b70a76d6-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wchmh\" (UID: \"7e4c0879-52a1-4038-aedd-3810b70a76d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"
Nov 03 09:37:06.325883 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:06.325779    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm8xh\" (UniqueName: \"kubernetes.io/projected/7e4c0879-52a1-4038-aedd-3810b70a76d6-kube-api-access-cm8xh\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wchmh\" (UID: \"7e4c0879-52a1-4038-aedd-3810b70a76d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"
Nov 03 09:37:06.325883 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:06.325818    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0879-52a1-4038-aedd-3810b70a76d6-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wchmh\" (UID: \"7e4c0879-52a1-4038-aedd-3810b70a76d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"
Nov 03 09:37:06.325883 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:06.325842    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0879-52a1-4038-aedd-3810b70a76d6-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wchmh\" (UID: \"7e4c0879-52a1-4038-aedd-3810b70a76d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"
Nov 03 09:37:06.326382 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:06.325920    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0879-52a1-4038-aedd-3810b70a76d6-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wchmh\" (UID: \"7e4c0879-52a1-4038-aedd-3810b70a76d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"
Nov 03 09:37:06.326382 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:06.326054    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0879-52a1-4038-aedd-3810b70a76d6-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wchmh\" (UID: \"7e4c0879-52a1-4038-aedd-3810b70a76d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"
Nov 03 09:37:06.326382 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:06.326273    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0879-52a1-4038-aedd-3810b70a76d6-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wchmh\" (UID: \"7e4c0879-52a1-4038-aedd-3810b70a76d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"
Nov 03 09:37:06.326382 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:06.326375    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e4c0879-52a1-4038-aedd-3810b70a76d6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wchmh\" (UID: \"7e4c0879-52a1-4038-aedd-3810b70a76d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"
Nov 03 09:37:06.328163 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:06.328140    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0879-52a1-4038-aedd-3810b70a76d6-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wchmh\" (UID: \"7e4c0879-52a1-4038-aedd-3810b70a76d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"
Nov 03 09:37:06.328216 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:06.328179    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e4c0879-52a1-4038-aedd-3810b70a76d6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wchmh\" (UID: \"7e4c0879-52a1-4038-aedd-3810b70a76d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"
Nov 03 09:37:06.360834 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:06.360808    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"]
Nov 03 09:37:06.385896 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:06.385863    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e4c0879-52a1-4038-aedd-3810b70a76d6-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wchmh\" (UID: \"7e4c0879-52a1-4038-aedd-3810b70a76d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"
Nov 03 09:37:06.386120 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:06.385918    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm8xh\" (UniqueName: \"kubernetes.io/projected/7e4c0879-52a1-4038-aedd-3810b70a76d6-kube-api-access-cm8xh\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-wchmh\" (UID: \"7e4c0879-52a1-4038-aedd-3810b70a76d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"
Nov 03 09:37:06.475483 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:06.475409    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"
Nov 03 09:37:06.905887 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:06.905856    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"]
Nov 03 09:37:06.906883 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:37:06.906857    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e4c0879_52a1_4038_aedd_3810b70a76d6.slice/crio-7f8fdf7b6459407f8f3be4dd2c44f28f702d03311b65ff458272e392a0eef045 WatchSource:0}: Error finding container 7f8fdf7b6459407f8f3be4dd2c44f28f702d03311b65ff458272e392a0eef045: Status 404 returned error can't find the container with id 7f8fdf7b6459407f8f3be4dd2c44f28f702d03311b65ff458272e392a0eef045
Nov 03 09:37:06.910035 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:06.910001    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:37:06.910155 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:06.910070    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:37:06.910155 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:06.910101    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:37:07.159554 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:07.159449    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh" event={"ID":"7e4c0879-52a1-4038-aedd-3810b70a76d6","Type":"ContainerStarted","Data":"c05c2fa8c1d5bd139ecf0972bd2afd405dbb16727f6978bd7d92260ff27bf6ab"}
Nov 03 09:37:07.159554 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:07.159517    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh" event={"ID":"7e4c0879-52a1-4038-aedd-3810b70a76d6","Type":"ContainerStarted","Data":"7f8fdf7b6459407f8f3be4dd2c44f28f702d03311b65ff458272e392a0eef045"}
Nov 03 09:37:07.279525 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:07.279456    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh" podStartSLOduration=1.279436886 podStartE2EDuration="1.279436886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:37:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:37:07.278647362 +0000 UTC m=+3451.547710829" watchObservedRunningTime="2026-11-03 09:37:07.279436886 +0000 UTC m=+3451.548500327"
Nov 03 09:37:07.478928 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:07.478835    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"
Nov 03 09:37:07.480775 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:07.480644    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-wchmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Nov 03 09:37:07.480775 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:07.480711    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh" podUID="7e4c0879-52a1-4038-aedd-3810b70a76d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Nov 03 09:37:08.476047 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:08.476016    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-wchmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Nov 03 09:37:08.476231 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:08.476064    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh" podUID="7e4c0879-52a1-4038-aedd-3810b70a76d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Nov 03 09:37:09.476294 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:09.476262    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-wchmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Nov 03 09:37:09.476695 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:09.476329    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh" podUID="7e4c0879-52a1-4038-aedd-3810b70a76d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Nov 03 09:37:10.475847 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:10.475809    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-wchmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Nov 03 09:37:10.476094 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:10.475865    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh" podUID="7e4c0879-52a1-4038-aedd-3810b70a76d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Nov 03 09:37:11.476622 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:11.476588    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-wchmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Nov 03 09:37:11.477053 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:11.476638    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh" podUID="7e4c0879-52a1-4038-aedd-3810b70a76d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Nov 03 09:37:12.476557 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:12.476524    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-wchmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Nov 03 09:37:12.476726 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:12.476588    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh" podUID="7e4c0879-52a1-4038-aedd-3810b70a76d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Nov 03 09:37:13.480413 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:13.480377    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-wchmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Nov 03 09:37:13.481079 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:13.481025    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh" podUID="7e4c0879-52a1-4038-aedd-3810b70a76d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Nov 03 09:37:14.476361 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:14.476325    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-wchmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Nov 03 09:37:14.476541 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:14.476379    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh" podUID="7e4c0879-52a1-4038-aedd-3810b70a76d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Nov 03 09:37:15.793378 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:15.793293    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"
Nov 03 09:37:15.794098 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:15.794074    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"
Nov 03 09:37:15.795085 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:15.795063    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"
Nov 03 09:37:16.154302 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:16.154266    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-373116-759d64f4b7-r4dmq"
Nov 03 09:37:16.154712 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:16.154681    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-373116-5645958765-96jxp"
Nov 03 09:37:16.938153 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:37:16.938068    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:37:21.277129 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:21.277093    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"]
Nov 03 09:37:21.281087 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:21.281062    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"
Nov 03 09:37:21.338658 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:21.338631    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jfwbt\" (UID: \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"
Nov 03 09:37:21.338791 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:21.338663    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jfwbt\" (UID: \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"
Nov 03 09:37:21.338791 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:21.338689    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jfwbt\" (UID: \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"
Nov 03 09:37:21.338870 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:21.338841    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jfwbt\" (UID: \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"
Nov 03 09:37:21.338911 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:21.338902    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcsrl\" (UniqueName: \"kubernetes.io/projected/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-kube-api-access-hcsrl\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jfwbt\" (UID: \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"
Nov 03 09:37:21.339005 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:21.338992    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jfwbt\" (UID: \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"
Nov 03 09:37:21.339057 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:21.339013    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jfwbt\" (UID: \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"
Nov 03 09:37:21.339117 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:21.339062    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jfwbt\" (UID: \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"
Nov 03 09:37:21.339200 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:21.339176    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jfwbt\" (UID: \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"
Nov 03 09:37:21.440434 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:21.440399    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jfwbt\" (UID: \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"
Nov 03 09:37:21.440601 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:21.440463    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jfwbt\" (UID: \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"
Nov 03 09:37:21.440601 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:21.440499    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jfwbt\" (UID: \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"
Nov 03 09:37:21.440601 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:21.440573    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jfwbt\" (UID: \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"
Nov 03 09:37:21.440790 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:21.440622    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcsrl\" (UniqueName: \"kubernetes.io/projected/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-kube-api-access-hcsrl\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jfwbt\" (UID: \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"
Nov 03 09:37:21.440790 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:21.440777    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jfwbt\" (UID: \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"
Nov 03 09:37:21.440913 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:21.440814    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jfwbt\" (UID: \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"
Nov 03 09:37:21.440913 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:21.440897    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jfwbt\" (UID: \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"
Nov 03 09:37:21.441435 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:21.440922    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jfwbt\" (UID: \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"
Nov 03 09:37:21.441435 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:21.440936    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jfwbt\" (UID: \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"
Nov 03 09:37:21.441435 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:21.440987    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jfwbt\" (UID: \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"
Nov 03 09:37:21.441435 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:21.440852    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jfwbt\" (UID: \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"
Nov 03 09:37:21.441435 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:21.441359    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jfwbt\" (UID: \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"
Nov 03 09:37:21.441692 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:21.441664    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jfwbt\" (UID: \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"
Nov 03 09:37:21.443465 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:21.443438    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jfwbt\" (UID: \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"
Nov 03 09:37:21.444158 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:21.444135    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jfwbt\" (UID: \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"
Nov 03 09:37:21.485665 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:21.485627    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"]
Nov 03 09:37:21.499849 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:21.499819    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jfwbt\" (UID: \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"
Nov 03 09:37:21.499990 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:21.499929    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcsrl\" (UniqueName: \"kubernetes.io/projected/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-kube-api-access-hcsrl\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jfwbt\" (UID: \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"
Nov 03 09:37:21.595568 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:21.595507    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"
Nov 03 09:37:22.024016 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:22.023983    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"]
Nov 03 09:37:22.025171 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:37:22.025145    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a6ab0a4_33a9_43f4_975b_2e3a6261e300.slice/crio-f9ef2add721f1182d8b32f7407ec37583b16408bc6b151a67458749f408514f8 WatchSource:0}: Error finding container f9ef2add721f1182d8b32f7407ec37583b16408bc6b151a67458749f408514f8: Status 404 returned error can't find the container with id f9ef2add721f1182d8b32f7407ec37583b16408bc6b151a67458749f408514f8
Nov 03 09:37:22.027877 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:22.027843    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:37:22.027987 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:22.027908    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:37:22.027987 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:22.027936    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:37:22.284599 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:22.284524    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt" event={"ID":"5a6ab0a4-33a9-43f4-975b-2e3a6261e300","Type":"ContainerStarted","Data":"2127eea8f47096b95545a7a0219a195ae3bd8f1fab1eefe055e519fead578408"}
Nov 03 09:37:22.284599 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:22.284572    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt" event={"ID":"5a6ab0a4-33a9-43f4-975b-2e3a6261e300","Type":"ContainerStarted","Data":"f9ef2add721f1182d8b32f7407ec37583b16408bc6b151a67458749f408514f8"}
Nov 03 09:37:22.416015 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:22.411156    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt" podStartSLOduration=1.411136683 podStartE2EDuration="1.411136683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:37:22.409643468 +0000 UTC m=+3466.678706925" watchObservedRunningTime="2026-11-03 09:37:22.411136683 +0000 UTC m=+3466.680200118"
Nov 03 09:37:22.596181 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:22.596087    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"
Nov 03 09:37:23.597761 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:23.597730    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Nov 03 09:37:23.598216 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:23.597796    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt" podUID="5a6ab0a4-33a9-43f4-975b-2e3a6261e300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Nov 03 09:37:24.596749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:24.596712    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Nov 03 09:37:24.596959 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:24.596767    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt" podUID="5a6ab0a4-33a9-43f4-975b-2e3a6261e300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Nov 03 09:37:25.596412 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:25.596378    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Nov 03 09:37:25.596829 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:25.596441    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt" podUID="5a6ab0a4-33a9-43f4-975b-2e3a6261e300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Nov 03 09:37:26.596739 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:26.596704    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Nov 03 09:37:26.597184 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:26.596763    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt" podUID="5a6ab0a4-33a9-43f4-975b-2e3a6261e300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Nov 03 09:37:27.596168 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:27.596132    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Nov 03 09:37:27.596361 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:27.596201    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt" podUID="5a6ab0a4-33a9-43f4-975b-2e3a6261e300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Nov 03 09:37:28.596590 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:28.596548    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Nov 03 09:37:28.597011 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:28.596615    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt" podUID="5a6ab0a4-33a9-43f4-975b-2e3a6261e300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Nov 03 09:37:29.596509 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:29.596472    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Nov 03 09:37:29.596737 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:29.596527    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt" podUID="5a6ab0a4-33a9-43f4-975b-2e3a6261e300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Nov 03 09:37:30.881391 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:30.881360    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"
Nov 03 09:37:30.882109 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:30.882089    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"
Nov 03 09:37:30.883184 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:30.883166    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"
Nov 03 09:37:38.416486 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:38.416447    4473 scope.go:122] "RemoveContainer" containerID="c362c59d5cc356724d48183077cd1b1c57e30c7c21d63f678264859f3a854dd0"
Nov 03 09:37:38.425921 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:38.425900    4473 scope.go:122] "RemoveContainer" containerID="76fe69b20cbafe93f69cacafd299318b5a138a922d621ed67dc7890511d05457"
Nov 03 09:37:38.706321 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:38.706242    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-872281-5d9f945fc-wnl4l"]
Nov 03 09:37:38.711785 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:38.711768    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-872281-5d9f945fc-wnl4l"
Nov 03 09:37:38.797823 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:38.783673    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-872281-5d9f945fc-wnl4l"]
Nov 03 09:37:38.816195 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:38.816146    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-872281\""
Nov 03 09:37:38.816195 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:38.816197    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-872281-dockercfg-hknwz\""
Nov 03 09:37:38.816586 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:38.816319    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78sk4\" (UniqueName: \"kubernetes.io/projected/c4e5a6e1-68ac-4090-805f-e0a944f505b9-kube-api-access-78sk4\") pod \"payload-processing-e2e-inf-a-872281-5d9f945fc-wnl4l\" (UID: \"c4e5a6e1-68ac-4090-805f-e0a944f505b9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-872281-5d9f945fc-wnl4l"
Nov 03 09:37:38.816586 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:38.816387    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e5a6e1-68ac-4090-805f-e0a944f505b9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-872281-5d9f945fc-wnl4l\" (UID: \"c4e5a6e1-68ac-4090-805f-e0a944f505b9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-872281-5d9f945fc-wnl4l"
Nov 03 09:37:38.899796 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:38.899753    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-872281-546f59c95-7tpjd"]
Nov 03 09:37:38.903736 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:38.903718    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872281-546f59c95-7tpjd"
Nov 03 09:37:38.917610 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:38.917584    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78sk4\" (UniqueName: \"kubernetes.io/projected/c4e5a6e1-68ac-4090-805f-e0a944f505b9-kube-api-access-78sk4\") pod \"payload-processing-e2e-inf-a-872281-5d9f945fc-wnl4l\" (UID: \"c4e5a6e1-68ac-4090-805f-e0a944f505b9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-872281-5d9f945fc-wnl4l"
Nov 03 09:37:38.917875 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:38.917847    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e5a6e1-68ac-4090-805f-e0a944f505b9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-872281-5d9f945fc-wnl4l\" (UID: \"c4e5a6e1-68ac-4090-805f-e0a944f505b9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-872281-5d9f945fc-wnl4l"
Nov 03 09:37:38.918545 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:38.918522    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e5a6e1-68ac-4090-805f-e0a944f505b9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-872281-5d9f945fc-wnl4l\" (UID: \"c4e5a6e1-68ac-4090-805f-e0a944f505b9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-872281-5d9f945fc-wnl4l"
Nov 03 09:37:38.961544 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:38.961472    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-872281-546f59c95-7tpjd"]
Nov 03 09:37:38.980873 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:38.980837    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78sk4\" (UniqueName: \"kubernetes.io/projected/c4e5a6e1-68ac-4090-805f-e0a944f505b9-kube-api-access-78sk4\") pod \"payload-processing-e2e-inf-a-872281-5d9f945fc-wnl4l\" (UID: \"c4e5a6e1-68ac-4090-805f-e0a944f505b9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-872281-5d9f945fc-wnl4l"
Nov 03 09:37:39.018723 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:39.018677    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/341adb1c-5c4f-41ce-a0e4-d929d8b32121-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-872281-546f59c95-7tpjd\" (UID: \"341adb1c-5c4f-41ce-a0e4-d929d8b32121\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872281-546f59c95-7tpjd"
Nov 03 09:37:39.018857 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:39.018820    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9s42\" (UniqueName: \"kubernetes.io/projected/341adb1c-5c4f-41ce-a0e4-d929d8b32121-kube-api-access-d9s42\") pod \"payload-pre-processing-e2e-inf-a-872281-546f59c95-7tpjd\" (UID: \"341adb1c-5c4f-41ce-a0e4-d929d8b32121\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872281-546f59c95-7tpjd"
Nov 03 09:37:39.084711 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:39.084676    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-872281-5d9f945fc-wnl4l"
Nov 03 09:37:39.119755 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:39.119723    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/341adb1c-5c4f-41ce-a0e4-d929d8b32121-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-872281-546f59c95-7tpjd\" (UID: \"341adb1c-5c4f-41ce-a0e4-d929d8b32121\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872281-546f59c95-7tpjd"
Nov 03 09:37:39.119961 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:39.119827    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9s42\" (UniqueName: \"kubernetes.io/projected/341adb1c-5c4f-41ce-a0e4-d929d8b32121-kube-api-access-d9s42\") pod \"payload-pre-processing-e2e-inf-a-872281-546f59c95-7tpjd\" (UID: \"341adb1c-5c4f-41ce-a0e4-d929d8b32121\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872281-546f59c95-7tpjd"
Nov 03 09:37:39.120444 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:39.120422    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/341adb1c-5c4f-41ce-a0e4-d929d8b32121-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-872281-546f59c95-7tpjd\" (UID: \"341adb1c-5c4f-41ce-a0e4-d929d8b32121\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872281-546f59c95-7tpjd"
Nov 03 09:37:39.189694 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:39.189665    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9s42\" (UniqueName: \"kubernetes.io/projected/341adb1c-5c4f-41ce-a0e4-d929d8b32121-kube-api-access-d9s42\") pod \"payload-pre-processing-e2e-inf-a-872281-546f59c95-7tpjd\" (UID: \"341adb1c-5c4f-41ce-a0e4-d929d8b32121\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872281-546f59c95-7tpjd"
Nov 03 09:37:39.217485 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:39.217428    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872281-546f59c95-7tpjd"
Nov 03 09:37:39.462032 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:39.462011    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-872281-5d9f945fc-wnl4l"]
Nov 03 09:37:39.463377 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:37:39.463350    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4e5a6e1_68ac_4090_805f_e0a944f505b9.slice/crio-312b04ce7e3632dc4b323ff9eb33d6da13fe965403e09057430ade16513f71a7 WatchSource:0}: Error finding container 312b04ce7e3632dc4b323ff9eb33d6da13fe965403e09057430ade16513f71a7: Status 404 returned error can't find the container with id 312b04ce7e3632dc4b323ff9eb33d6da13fe965403e09057430ade16513f71a7
Nov 03 09:37:39.593146 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:39.593117    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-872281-546f59c95-7tpjd"]
Nov 03 09:37:39.593860 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:37:39.593839    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod341adb1c_5c4f_41ce_a0e4_d929d8b32121.slice/crio-7e1026e6eaa055550bbc29adf8eb18c22c5cd1d4ca8bb830b71972a2804365b6 WatchSource:0}: Error finding container 7e1026e6eaa055550bbc29adf8eb18c22c5cd1d4ca8bb830b71972a2804365b6: Status 404 returned error can't find the container with id 7e1026e6eaa055550bbc29adf8eb18c22c5cd1d4ca8bb830b71972a2804365b6
Nov 03 09:37:40.436033 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:40.435994    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872281-546f59c95-7tpjd" event={"ID":"341adb1c-5c4f-41ce-a0e4-d929d8b32121","Type":"ContainerStarted","Data":"33463b1eb9bc5179227ce4525a53f2b38ab431eb09fe2d60c57dfce5eac627f1"}
Nov 03 09:37:40.436268 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:40.436064    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872281-546f59c95-7tpjd" event={"ID":"341adb1c-5c4f-41ce-a0e4-d929d8b32121","Type":"ContainerStarted","Data":"7e1026e6eaa055550bbc29adf8eb18c22c5cd1d4ca8bb830b71972a2804365b6"}
Nov 03 09:37:40.436268 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:40.436122    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872281-546f59c95-7tpjd"
Nov 03 09:37:40.437657 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:40.437634    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-872281-5d9f945fc-wnl4l" event={"ID":"c4e5a6e1-68ac-4090-805f-e0a944f505b9","Type":"ContainerStarted","Data":"8d93dd4061a6eb49b43563f03a4d294ece33590d5356fdd122c70156da14da23"}
Nov 03 09:37:40.437781 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:40.437675    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-872281-5d9f945fc-wnl4l"
Nov 03 09:37:40.437781 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:40.437695    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-872281-5d9f945fc-wnl4l" event={"ID":"c4e5a6e1-68ac-4090-805f-e0a944f505b9","Type":"ContainerStarted","Data":"312b04ce7e3632dc4b323ff9eb33d6da13fe965403e09057430ade16513f71a7"}
Nov 03 09:37:40.553057 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:40.553009    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872281-546f59c95-7tpjd" podStartSLOduration=2.552996232 podStartE2EDuration="2.552996232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:37:40.551059865 +0000 UTC m=+3484.820123344" watchObservedRunningTime="2026-11-03 09:37:40.552996232 +0000 UTC m=+3484.822059667"
Nov 03 09:37:40.693849 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:40.693729    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-872281-5d9f945fc-wnl4l" podStartSLOduration=2.693704123 podStartE2EDuration="2.693704123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:37:40.67363938 +0000 UTC m=+3484.942702853" watchObservedRunningTime="2026-11-03 09:37:40.693704123 +0000 UTC m=+3484.962767558"
Nov 03 09:37:42.055833 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:42.055791    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5adb63-7767f95b67-5kjcm"]
Nov 03 09:37:46.937877 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:37:46.937843    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:37:47.056186 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:47.056149    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5adb63-7767f95b67-5kjcm" podUID="c2375513-57e5-432f-88a5-c61bb7fad533" containerName="payload-pre-processing" containerID="cri-o://96b89d3fae8eaeec9992fcc523444eaae79deacc6587b3877ceed568b91a3b5c" gracePeriod=25
Nov 03 09:37:47.316435 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:47.316414    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5adb63-7767f95b67-5kjcm"
Nov 03 09:37:47.421136 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:47.421105    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2375513-57e5-432f-88a5-c61bb7fad533-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2375513-57e5-432f-88a5-c61bb7fad533-plugins-config-volume\") pod \"c2375513-57e5-432f-88a5-c61bb7fad533\" (UID: \"c2375513-57e5-432f-88a5-c61bb7fad533\") "
Nov 03 09:37:47.421349 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:47.421280    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2375513-57e5-432f-88a5-c61bb7fad533-kube-api-access-ql9ld\" (UniqueName: \"kubernetes.io/projected/c2375513-57e5-432f-88a5-c61bb7fad533-kube-api-access-ql9ld\") pod \"c2375513-57e5-432f-88a5-c61bb7fad533\" (UID: \"c2375513-57e5-432f-88a5-c61bb7fad533\") "
Nov 03 09:37:47.421522 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:47.421500    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2375513-57e5-432f-88a5-c61bb7fad533-plugins-config-volume" pod "c2375513-57e5-432f-88a5-c61bb7fad533" (UID: "c2375513-57e5-432f-88a5-c61bb7fad533"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:37:47.421701 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:47.421683    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2375513-57e5-432f-88a5-c61bb7fad533-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:37:47.423851 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:47.423825    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2375513-57e5-432f-88a5-c61bb7fad533-kube-api-access-ql9ld" pod "c2375513-57e5-432f-88a5-c61bb7fad533" (UID: "c2375513-57e5-432f-88a5-c61bb7fad533"). InnerVolumeSpecName "kube-api-access-ql9ld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:37:47.482622 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:47.482546    4473 generic.go:356] "Generic (PLEG): container finished" podID="c2375513-57e5-432f-88a5-c61bb7fad533" containerID="96b89d3fae8eaeec9992fcc523444eaae79deacc6587b3877ceed568b91a3b5c" exitCode=0
Nov 03 09:37:47.482622 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:47.482593    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5adb63-7767f95b67-5kjcm"
Nov 03 09:37:47.482622 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:47.482585    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5adb63-7767f95b67-5kjcm" event={"ID":"c2375513-57e5-432f-88a5-c61bb7fad533","Type":"ContainerDied","Data":"96b89d3fae8eaeec9992fcc523444eaae79deacc6587b3877ceed568b91a3b5c"}
Nov 03 09:37:47.482822 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:47.482657    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5adb63-7767f95b67-5kjcm" event={"ID":"c2375513-57e5-432f-88a5-c61bb7fad533","Type":"ContainerDied","Data":"546c6e76f91abae5a20e14b95d83368cdd3f0174e4f025b7b3324686a0036b4c"}
Nov 03 09:37:47.482822 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:47.482684    4473 scope.go:122] "RemoveContainer" containerID="96b89d3fae8eaeec9992fcc523444eaae79deacc6587b3877ceed568b91a3b5c"
Nov 03 09:37:47.495076 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:47.494919    4473 scope.go:122] "RemoveContainer" containerID="96b89d3fae8eaeec9992fcc523444eaae79deacc6587b3877ceed568b91a3b5c"
Nov 03 09:37:47.495261 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:37:47.495240    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96b89d3fae8eaeec9992fcc523444eaae79deacc6587b3877ceed568b91a3b5c\": container with ID starting with 96b89d3fae8eaeec9992fcc523444eaae79deacc6587b3877ceed568b91a3b5c not found: ID does not exist" containerID="96b89d3fae8eaeec9992fcc523444eaae79deacc6587b3877ceed568b91a3b5c"
Nov 03 09:37:47.495331 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:47.495268    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96b89d3fae8eaeec9992fcc523444eaae79deacc6587b3877ceed568b91a3b5c"} err="failed to get container status \"96b89d3fae8eaeec9992fcc523444eaae79deacc6587b3877ceed568b91a3b5c\": rpc error: code = NotFound desc = could not find container \"96b89d3fae8eaeec9992fcc523444eaae79deacc6587b3877ceed568b91a3b5c\": container with ID starting with 96b89d3fae8eaeec9992fcc523444eaae79deacc6587b3877ceed568b91a3b5c not found: ID does not exist"
Nov 03 09:37:47.522399 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:47.522377    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql9ld\" (UniqueName: \"kubernetes.io/projected/c2375513-57e5-432f-88a5-c61bb7fad533-kube-api-access-ql9ld\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:37:47.653317 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:47.653287    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5adb63-7767f95b67-5kjcm"]
Nov 03 09:37:47.658162 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:47.658134    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-5adb63-7767f95b67-5kjcm"]
Nov 03 09:37:47.870332 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:47.870298    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2375513-57e5-432f-88a5-c61bb7fad533" path="/var/lib/kubelet/pods/c2375513-57e5-432f-88a5-c61bb7fad533/volumes"
Nov 03 09:37:51.450161 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:51.450133    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-872281-5d9f945fc-wnl4l"
Nov 03 09:37:51.450640 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:51.450205    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872281-546f59c95-7tpjd"
Nov 03 09:37:52.264932 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:52.264891    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"]
Nov 03 09:37:52.265147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:52.265117    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd" podUID="f562628b-a92b-4022-8fe9-4f65cc2f11bb" containerName="istio-proxy" containerID="cri-o://4893a04622108de79d791a5e6a1394ffb2bc0151a613a4bdca0c02ca63712696" gracePeriod=30
Nov 03 09:37:57.530165 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.530105    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"
Nov 03 09:37:57.546004 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.545979    4473 generic.go:356] "Generic (PLEG): container finished" podID="f562628b-a92b-4022-8fe9-4f65cc2f11bb" containerID="4893a04622108de79d791a5e6a1394ffb2bc0151a613a4bdca0c02ca63712696" exitCode=0
Nov 03 09:37:57.546125 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.546037    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd" event={"ID":"f562628b-a92b-4022-8fe9-4f65cc2f11bb","Type":"ContainerDied","Data":"4893a04622108de79d791a5e6a1394ffb2bc0151a613a4bdca0c02ca63712696"}
Nov 03 09:37:57.546125 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.546076    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd" event={"ID":"f562628b-a92b-4022-8fe9-4f65cc2f11bb","Type":"ContainerDied","Data":"bf260cdbd8856c7fef82a3dbdeb242bafe7850ada1569afa7c025a61882aa747"}
Nov 03 09:37:57.546125 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.546079    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"
Nov 03 09:37:57.546125 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.546098    4473 scope.go:122] "RemoveContainer" containerID="4893a04622108de79d791a5e6a1394ffb2bc0151a613a4bdca0c02ca63712696"
Nov 03 09:37:57.555602 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.555586    4473 scope.go:122] "RemoveContainer" containerID="33a4ac9fad5183ab16fc816b3a44ce89b2738c7ba18228888e578e0fc6d3d6b9"
Nov 03 09:37:57.564180 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.564161    4473 scope.go:122] "RemoveContainer" containerID="4893a04622108de79d791a5e6a1394ffb2bc0151a613a4bdca0c02ca63712696"
Nov 03 09:37:57.564410 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:37:57.564393    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4893a04622108de79d791a5e6a1394ffb2bc0151a613a4bdca0c02ca63712696\": container with ID starting with 4893a04622108de79d791a5e6a1394ffb2bc0151a613a4bdca0c02ca63712696 not found: ID does not exist" containerID="4893a04622108de79d791a5e6a1394ffb2bc0151a613a4bdca0c02ca63712696"
Nov 03 09:37:57.564458 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.564419    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4893a04622108de79d791a5e6a1394ffb2bc0151a613a4bdca0c02ca63712696"} err="failed to get container status \"4893a04622108de79d791a5e6a1394ffb2bc0151a613a4bdca0c02ca63712696\": rpc error: code = NotFound desc = could not find container \"4893a04622108de79d791a5e6a1394ffb2bc0151a613a4bdca0c02ca63712696\": container with ID starting with 4893a04622108de79d791a5e6a1394ffb2bc0151a613a4bdca0c02ca63712696 not found: ID does not exist"
Nov 03 09:37:57.564458 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.564435    4473 scope.go:122] "RemoveContainer" containerID="33a4ac9fad5183ab16fc816b3a44ce89b2738c7ba18228888e578e0fc6d3d6b9"
Nov 03 09:37:57.564696 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:37:57.564671    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33a4ac9fad5183ab16fc816b3a44ce89b2738c7ba18228888e578e0fc6d3d6b9\": container with ID starting with 33a4ac9fad5183ab16fc816b3a44ce89b2738c7ba18228888e578e0fc6d3d6b9 not found: ID does not exist" containerID="33a4ac9fad5183ab16fc816b3a44ce89b2738c7ba18228888e578e0fc6d3d6b9"
Nov 03 09:37:57.564748 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.564702    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33a4ac9fad5183ab16fc816b3a44ce89b2738c7ba18228888e578e0fc6d3d6b9"} err="failed to get container status \"33a4ac9fad5183ab16fc816b3a44ce89b2738c7ba18228888e578e0fc6d3d6b9\": rpc error: code = NotFound desc = could not find container \"33a4ac9fad5183ab16fc816b3a44ce89b2738c7ba18228888e578e0fc6d3d6b9\": container with ID starting with 33a4ac9fad5183ab16fc816b3a44ce89b2738c7ba18228888e578e0fc6d3d6b9 not found: ID does not exist"
Nov 03 09:37:57.630930 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.630908    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f562628b-a92b-4022-8fe9-4f65cc2f11bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f562628b-a92b-4022-8fe9-4f65cc2f11bb-istio-envoy\") pod \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\" (UID: \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\") "
Nov 03 09:37:57.631044 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.631014    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f562628b-a92b-4022-8fe9-4f65cc2f11bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f562628b-a92b-4022-8fe9-4f65cc2f11bb-istiod-ca-cert\") pod \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\" (UID: \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\") "
Nov 03 09:37:57.631089 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.631060    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f562628b-a92b-4022-8fe9-4f65cc2f11bb-kube-api-access-qqnfj\" (UniqueName: \"kubernetes.io/projected/f562628b-a92b-4022-8fe9-4f65cc2f11bb-kube-api-access-qqnfj\") pod \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\" (UID: \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\") "
Nov 03 09:37:57.631133 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.631116    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f562628b-a92b-4022-8fe9-4f65cc2f11bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f562628b-a92b-4022-8fe9-4f65cc2f11bb-workload-certs\") pod \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\" (UID: \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\") "
Nov 03 09:37:57.631188 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.631140    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f562628b-a92b-4022-8fe9-4f65cc2f11bb-istio-token\" (UniqueName: \"kubernetes.io/projected/f562628b-a92b-4022-8fe9-4f65cc2f11bb-istio-token\") pod \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\" (UID: \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\") "
Nov 03 09:37:57.631188 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.631159    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f562628b-a92b-4022-8fe9-4f65cc2f11bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f562628b-a92b-4022-8fe9-4f65cc2f11bb-credential-socket\") pod \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\" (UID: \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\") "
Nov 03 09:37:57.631301 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.631195    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f562628b-a92b-4022-8fe9-4f65cc2f11bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f562628b-a92b-4022-8fe9-4f65cc2f11bb-istio-data\") pod \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\" (UID: \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\") "
Nov 03 09:37:57.631301 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.631266    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f562628b-a92b-4022-8fe9-4f65cc2f11bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f562628b-a92b-4022-8fe9-4f65cc2f11bb-workload-socket\") pod \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\" (UID: \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\") "
Nov 03 09:37:57.631413 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.631301    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f562628b-a92b-4022-8fe9-4f65cc2f11bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f562628b-a92b-4022-8fe9-4f65cc2f11bb-istio-podinfo\") pod \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\" (UID: \"f562628b-a92b-4022-8fe9-4f65cc2f11bb\") "
Nov 03 09:37:57.631468 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.631425    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f562628b-a92b-4022-8fe9-4f65cc2f11bb-istiod-ca-cert" pod "f562628b-a92b-4022-8fe9-4f65cc2f11bb" (UID: "f562628b-a92b-4022-8fe9-4f65cc2f11bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:37:57.631468 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.631427    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f562628b-a92b-4022-8fe9-4f65cc2f11bb-workload-certs" pod "f562628b-a92b-4022-8fe9-4f65cc2f11bb" (UID: "f562628b-a92b-4022-8fe9-4f65cc2f11bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:37:57.631468 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.631450    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f562628b-a92b-4022-8fe9-4f65cc2f11bb-credential-socket" pod "f562628b-a92b-4022-8fe9-4f65cc2f11bb" (UID: "f562628b-a92b-4022-8fe9-4f65cc2f11bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:37:57.631655 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.631634    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f562628b-a92b-4022-8fe9-4f65cc2f11bb-workload-socket" pod "f562628b-a92b-4022-8fe9-4f65cc2f11bb" (UID: "f562628b-a92b-4022-8fe9-4f65cc2f11bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:37:57.631773 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.631679    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f562628b-a92b-4022-8fe9-4f65cc2f11bb-istio-data" pod "f562628b-a92b-4022-8fe9-4f65cc2f11bb" (UID: "f562628b-a92b-4022-8fe9-4f65cc2f11bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:37:57.631857 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.631840    4473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f562628b-a92b-4022-8fe9-4f65cc2f11bb-workload-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:37:57.631915 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.631860    4473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f562628b-a92b-4022-8fe9-4f65cc2f11bb-credential-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:37:57.631915 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.631869    4473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f562628b-a92b-4022-8fe9-4f65cc2f11bb-istio-data\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:37:57.631915 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.631878    4473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f562628b-a92b-4022-8fe9-4f65cc2f11bb-workload-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:37:57.631915 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.631887    4473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f562628b-a92b-4022-8fe9-4f65cc2f11bb-istiod-ca-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:37:57.633364 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.633338    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f562628b-a92b-4022-8fe9-4f65cc2f11bb-istio-token" pod "f562628b-a92b-4022-8fe9-4f65cc2f11bb" (UID: "f562628b-a92b-4022-8fe9-4f65cc2f11bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:37:57.633629 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.633607    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f562628b-a92b-4022-8fe9-4f65cc2f11bb-istio-podinfo" pod "f562628b-a92b-4022-8fe9-4f65cc2f11bb" (UID: "f562628b-a92b-4022-8fe9-4f65cc2f11bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 09:37:57.633794 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.633774    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f562628b-a92b-4022-8fe9-4f65cc2f11bb-istio-envoy" pod "f562628b-a92b-4022-8fe9-4f65cc2f11bb" (UID: "f562628b-a92b-4022-8fe9-4f65cc2f11bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:37:57.633794 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.633782    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f562628b-a92b-4022-8fe9-4f65cc2f11bb-kube-api-access-qqnfj" pod "f562628b-a92b-4022-8fe9-4f65cc2f11bb" (UID: "f562628b-a92b-4022-8fe9-4f65cc2f11bb"). InnerVolumeSpecName "kube-api-access-qqnfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:37:57.732690 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.732648    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqnfj\" (UniqueName: \"kubernetes.io/projected/f562628b-a92b-4022-8fe9-4f65cc2f11bb-kube-api-access-qqnfj\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:37:57.732690 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.732676    4473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f562628b-a92b-4022-8fe9-4f65cc2f11bb-istio-token\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:37:57.732690 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.732692    4473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f562628b-a92b-4022-8fe9-4f65cc2f11bb-istio-podinfo\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:37:57.732908 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.732702    4473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f562628b-a92b-4022-8fe9-4f65cc2f11bb-istio-envoy\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:37:57.971539 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.971506    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"]
Nov 03 09:37:57.976769 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:57.976745    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5adb63-openshift-default-74bcb74df9-ngcrd"]
Nov 03 09:37:59.870160 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:37:59.870127    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f562628b-a92b-4022-8fe9-4f65cc2f11bb" path="/var/lib/kubelet/pods/f562628b-a92b-4022-8fe9-4f65cc2f11bb/volumes"
Nov 03 09:38:16.057702 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:16.057665    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:38:16.938178 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:38:16.938138    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:38:23.796980 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:23.792718    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ff6461-6884958899-5sml8"]
Nov 03 09:38:23.796980 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:23.793558    4473 cpu_manager.go:414] "containerMap: removing container" podUID="f562628b-a92b-4022-8fe9-4f65cc2f11bb" containerName="istio-proxy"
Nov 03 09:38:23.796980 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:23.793578    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f562628b-a92b-4022-8fe9-4f65cc2f11bb" containerName="istio-proxy"
Nov 03 09:38:23.796980 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:23.793604    4473 cpu_manager.go:414] "containerMap: removing container" podUID="c2375513-57e5-432f-88a5-c61bb7fad533" containerName="payload-pre-processing"
Nov 03 09:38:23.796980 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:23.793613    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2375513-57e5-432f-88a5-c61bb7fad533" containerName="payload-pre-processing"
Nov 03 09:38:23.796980 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:23.793784    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="f562628b-a92b-4022-8fe9-4f65cc2f11bb" containerName="istio-proxy"
Nov 03 09:38:23.796980 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:23.793803    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="f562628b-a92b-4022-8fe9-4f65cc2f11bb" containerName="istio-proxy"
Nov 03 09:38:23.796980 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:23.793816    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2375513-57e5-432f-88a5-c61bb7fad533" containerName="payload-pre-processing"
Nov 03 09:38:23.823295 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:23.819348    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ff6461-6884958899-5sml8"
Nov 03 09:38:23.849194 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:23.849157    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j485n\" (UniqueName: \"kubernetes.io/projected/0c195d47-54ec-435a-9a6d-9f5d5caa5131-kube-api-access-j485n\") pod \"payload-processing-e2e-inf-a-ff6461-6884958899-5sml8\" (UID: \"0c195d47-54ec-435a-9a6d-9f5d5caa5131\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ff6461-6884958899-5sml8"
Nov 03 09:38:23.849522 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:23.849496    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c195d47-54ec-435a-9a6d-9f5d5caa5131-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ff6461-6884958899-5sml8\" (UID: \"0c195d47-54ec-435a-9a6d-9f5d5caa5131\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ff6461-6884958899-5sml8"
Nov 03 09:38:23.858284 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:23.854904    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ff6461-6884958899-5sml8"]
Nov 03 09:38:23.892411 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:23.892380    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ff6461\""
Nov 03 09:38:23.892588 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:23.892474    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ff6461-dockercfg-5fgj6\""
Nov 03 09:38:23.956915 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:23.956885    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ff6461-6ccb9d7b95-kklnx"]
Nov 03 09:38:23.957501 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:23.957485    4473 cpu_manager.go:414] "containerMap: removing container" podUID="f562628b-a92b-4022-8fe9-4f65cc2f11bb" containerName="istio-proxy"
Nov 03 09:38:23.957501 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:23.957503    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f562628b-a92b-4022-8fe9-4f65cc2f11bb" containerName="istio-proxy"
Nov 03 09:38:23.958954 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:23.958918    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j485n\" (UniqueName: \"kubernetes.io/projected/0c195d47-54ec-435a-9a6d-9f5d5caa5131-kube-api-access-j485n\") pod \"payload-processing-e2e-inf-a-ff6461-6884958899-5sml8\" (UID: \"0c195d47-54ec-435a-9a6d-9f5d5caa5131\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ff6461-6884958899-5sml8"
Nov 03 09:38:23.959015 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:23.958996    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c195d47-54ec-435a-9a6d-9f5d5caa5131-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ff6461-6884958899-5sml8\" (UID: \"0c195d47-54ec-435a-9a6d-9f5d5caa5131\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ff6461-6884958899-5sml8"
Nov 03 09:38:23.959663 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:23.959645    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c195d47-54ec-435a-9a6d-9f5d5caa5131-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ff6461-6884958899-5sml8\" (UID: \"0c195d47-54ec-435a-9a6d-9f5d5caa5131\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ff6461-6884958899-5sml8"
Nov 03 09:38:23.961158 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:23.961137    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff6461-6ccb9d7b95-kklnx"
Nov 03 09:38:24.019296 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:24.019265    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ff6461-6ccb9d7b95-kklnx"]
Nov 03 09:38:24.021718 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:24.021695    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j485n\" (UniqueName: \"kubernetes.io/projected/0c195d47-54ec-435a-9a6d-9f5d5caa5131-kube-api-access-j485n\") pod \"payload-processing-e2e-inf-a-ff6461-6884958899-5sml8\" (UID: \"0c195d47-54ec-435a-9a6d-9f5d5caa5131\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ff6461-6884958899-5sml8"
Nov 03 09:38:24.060728 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:24.060638    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ptsb\" (UniqueName: \"kubernetes.io/projected/a02abcb7-266e-40cd-a1fb-fca2b5706f14-kube-api-access-7ptsb\") pod \"payload-pre-processing-e2e-inf-a-ff6461-6ccb9d7b95-kklnx\" (UID: \"a02abcb7-266e-40cd-a1fb-fca2b5706f14\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff6461-6ccb9d7b95-kklnx"
Nov 03 09:38:24.060901 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:24.060763    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a02abcb7-266e-40cd-a1fb-fca2b5706f14-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ff6461-6ccb9d7b95-kklnx\" (UID: \"a02abcb7-266e-40cd-a1fb-fca2b5706f14\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff6461-6ccb9d7b95-kklnx"
Nov 03 09:38:24.162067 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:24.162035    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ff6461-6884958899-5sml8"
Nov 03 09:38:24.162397 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:24.162373    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ptsb\" (UniqueName: \"kubernetes.io/projected/a02abcb7-266e-40cd-a1fb-fca2b5706f14-kube-api-access-7ptsb\") pod \"payload-pre-processing-e2e-inf-a-ff6461-6ccb9d7b95-kklnx\" (UID: \"a02abcb7-266e-40cd-a1fb-fca2b5706f14\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff6461-6ccb9d7b95-kklnx"
Nov 03 09:38:24.162497 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:24.162480    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a02abcb7-266e-40cd-a1fb-fca2b5706f14-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ff6461-6ccb9d7b95-kklnx\" (UID: \"a02abcb7-266e-40cd-a1fb-fca2b5706f14\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff6461-6ccb9d7b95-kklnx"
Nov 03 09:38:24.163252 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:24.163233    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a02abcb7-266e-40cd-a1fb-fca2b5706f14-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ff6461-6ccb9d7b95-kklnx\" (UID: \"a02abcb7-266e-40cd-a1fb-fca2b5706f14\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff6461-6ccb9d7b95-kklnx"
Nov 03 09:38:24.222415 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:24.222384    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ptsb\" (UniqueName: \"kubernetes.io/projected/a02abcb7-266e-40cd-a1fb-fca2b5706f14-kube-api-access-7ptsb\") pod \"payload-pre-processing-e2e-inf-a-ff6461-6ccb9d7b95-kklnx\" (UID: \"a02abcb7-266e-40cd-a1fb-fca2b5706f14\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff6461-6ccb9d7b95-kklnx"
Nov 03 09:38:24.274085 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:24.274055    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff6461-6ccb9d7b95-kklnx"
Nov 03 09:38:24.545114 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:24.545078    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ff6461-6884958899-5sml8"]
Nov 03 09:38:24.545682 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:38:24.545656    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c195d47_54ec_435a_9a6d_9f5d5caa5131.slice/crio-35edfff3597010684b3a61be8ccc17115f9b18ff625f48d54128d119fddd8100 WatchSource:0}: Error finding container 35edfff3597010684b3a61be8ccc17115f9b18ff625f48d54128d119fddd8100: Status 404 returned error can't find the container with id 35edfff3597010684b3a61be8ccc17115f9b18ff625f48d54128d119fddd8100
Nov 03 09:38:24.652836 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:24.652810    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ff6461-6ccb9d7b95-kklnx"]
Nov 03 09:38:24.653811 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:38:24.653788    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda02abcb7_266e_40cd_a1fb_fca2b5706f14.slice/crio-a2b658990c0ac67a27f75aaaaa1afe968b43a0f95781f4e427a3df7b7106b9ec WatchSource:0}: Error finding container a2b658990c0ac67a27f75aaaaa1afe968b43a0f95781f4e427a3df7b7106b9ec: Status 404 returned error can't find the container with id a2b658990c0ac67a27f75aaaaa1afe968b43a0f95781f4e427a3df7b7106b9ec
Nov 03 09:38:24.724797 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:24.724763    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ff6461-6884958899-5sml8" event={"ID":"0c195d47-54ec-435a-9a6d-9f5d5caa5131","Type":"ContainerStarted","Data":"2591a2b76ecd6c6a3ea50c001b64a7603035b2dd5d285befdbf5e71a373ab884"}
Nov 03 09:38:24.724916 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:24.724831    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ff6461-6884958899-5sml8"
Nov 03 09:38:24.724916 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:24.724858    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ff6461-6884958899-5sml8" event={"ID":"0c195d47-54ec-435a-9a6d-9f5d5caa5131","Type":"ContainerStarted","Data":"35edfff3597010684b3a61be8ccc17115f9b18ff625f48d54128d119fddd8100"}
Nov 03 09:38:24.726661 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:24.726637    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff6461-6ccb9d7b95-kklnx" event={"ID":"a02abcb7-266e-40cd-a1fb-fca2b5706f14","Type":"ContainerStarted","Data":"066fb199953d3a91f831957ce59087fd3d47829e510e2e915bd1e10378675601"}
Nov 03 09:38:24.726771 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:24.726689    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff6461-6ccb9d7b95-kklnx" event={"ID":"a02abcb7-266e-40cd-a1fb-fca2b5706f14","Type":"ContainerStarted","Data":"a2b658990c0ac67a27f75aaaaa1afe968b43a0f95781f4e427a3df7b7106b9ec"}
Nov 03 09:38:24.726771 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:24.726719    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff6461-6ccb9d7b95-kklnx"
Nov 03 09:38:24.843891 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:24.843791    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-ff6461-6884958899-5sml8" podStartSLOduration=1.8437769419999999 podStartE2EDuration="1.843776942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:38:24.84043894 +0000 UTC m=+3529.109502375" watchObservedRunningTime="2026-11-03 09:38:24.843776942 +0000 UTC m=+3529.112840376"
Nov 03 09:38:24.964000 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:24.960468    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff6461-6ccb9d7b95-kklnx" podStartSLOduration=1.960448373 podStartE2EDuration="1.960448373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:38:24.959911968 +0000 UTC m=+3529.228975418" watchObservedRunningTime="2026-11-03 09:38:24.960448373 +0000 UTC m=+3529.229511808"
Nov 03 09:38:35.737061 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:35.737031    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff6461-6ccb9d7b95-kklnx"
Nov 03 09:38:35.737484 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:35.737348    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ff6461-6884958899-5sml8"
Nov 03 09:38:46.938456 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:38:46.938364    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:38:51.124870 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:38:51.124837    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:39:05.502111 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:05.502075    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42d4f8b1-6lr8w"]
Nov 03 09:39:05.506320 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:05.506305    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42d4f8b1-6lr8w"
Nov 03 09:39:05.565450 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:05.565421    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42d4f8b1-6lr8w"]
Nov 03 09:39:05.590345 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:05.590323    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/271b0cc5-8c5a-422d-b48f-85766ccc2731-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42d4f8b1-6lr8w\" (UID: \"271b0cc5-8c5a-422d-b48f-85766ccc2731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42d4f8b1-6lr8w"
Nov 03 09:39:05.590489 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:05.590358    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/271b0cc5-8c5a-422d-b48f-85766ccc2731-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42d4f8b1-6lr8w\" (UID: \"271b0cc5-8c5a-422d-b48f-85766ccc2731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42d4f8b1-6lr8w"
Nov 03 09:39:05.590560 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:05.590515    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqxcb\" (UniqueName: \"kubernetes.io/projected/271b0cc5-8c5a-422d-b48f-85766ccc2731-kube-api-access-zqxcb\") pod \"maas-api-revoke-sub-42d4f8b1-6lr8w\" (UID: \"271b0cc5-8c5a-422d-b48f-85766ccc2731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42d4f8b1-6lr8w"
Nov 03 09:39:05.611508 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:05.611487    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:39:05.611630 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:05.611518    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:39:05.611630 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:05.611564    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:39:05.611630 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:05.611613    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:39:05.691431 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:05.691405    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqxcb\" (UniqueName: \"kubernetes.io/projected/271b0cc5-8c5a-422d-b48f-85766ccc2731-kube-api-access-zqxcb\") pod \"maas-api-revoke-sub-42d4f8b1-6lr8w\" (UID: \"271b0cc5-8c5a-422d-b48f-85766ccc2731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42d4f8b1-6lr8w"
Nov 03 09:39:05.691593 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:05.691524    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/271b0cc5-8c5a-422d-b48f-85766ccc2731-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42d4f8b1-6lr8w\" (UID: \"271b0cc5-8c5a-422d-b48f-85766ccc2731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42d4f8b1-6lr8w"
Nov 03 09:39:05.691593 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:05.691546    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/271b0cc5-8c5a-422d-b48f-85766ccc2731-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42d4f8b1-6lr8w\" (UID: \"271b0cc5-8c5a-422d-b48f-85766ccc2731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42d4f8b1-6lr8w"
Nov 03 09:39:05.692200 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:05.692181    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/271b0cc5-8c5a-422d-b48f-85766ccc2731-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42d4f8b1-6lr8w\" (UID: \"271b0cc5-8c5a-422d-b48f-85766ccc2731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42d4f8b1-6lr8w"
Nov 03 09:39:05.694090 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:05.694069    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/271b0cc5-8c5a-422d-b48f-85766ccc2731-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42d4f8b1-6lr8w\" (UID: \"271b0cc5-8c5a-422d-b48f-85766ccc2731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42d4f8b1-6lr8w"
Nov 03 09:39:05.750585 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:05.750559    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqxcb\" (UniqueName: \"kubernetes.io/projected/271b0cc5-8c5a-422d-b48f-85766ccc2731-kube-api-access-zqxcb\") pod \"maas-api-revoke-sub-42d4f8b1-6lr8w\" (UID: \"271b0cc5-8c5a-422d-b48f-85766ccc2731\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42d4f8b1-6lr8w"
Nov 03 09:39:05.819701 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:05.819640    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42d4f8b1-6lr8w"
Nov 03 09:39:06.199643 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:06.199612    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42d4f8b1-6lr8w"]
Nov 03 09:39:06.200560 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:39:06.200540    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod271b0cc5_8c5a_422d_b48f_85766ccc2731.slice/crio-7716399461a2b9be88e514252e282713c279fcb92e7f31d6911c96101740ce31 WatchSource:0}: Error finding container 7716399461a2b9be88e514252e282713c279fcb92e7f31d6911c96101740ce31: Status 404 returned error can't find the container with id 7716399461a2b9be88e514252e282713c279fcb92e7f31d6911c96101740ce31
Nov 03 09:39:07.051967 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:07.051905    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42d4f8b1-6lr8w" event={"ID":"271b0cc5-8c5a-422d-b48f-85766ccc2731","Type":"ContainerStarted","Data":"4bd74d9920f6769412c06986d427f54026477900a8cd24dc5ced195dcb234b1a"}
Nov 03 09:39:07.052392 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:07.051987    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42d4f8b1-6lr8w" event={"ID":"271b0cc5-8c5a-422d-b48f-85766ccc2731","Type":"ContainerStarted","Data":"7716399461a2b9be88e514252e282713c279fcb92e7f31d6911c96101740ce31"}
Nov 03 09:39:07.171222 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:07.171169    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42d4f8b1-6lr8w" podStartSLOduration=2.171155055 podStartE2EDuration="2.171155055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:39:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:39:07.168967023 +0000 UTC m=+3571.438030457" watchObservedRunningTime="2026-11-03 09:39:07.171155055 +0000 UTC m=+3571.440218489"
Nov 03 09:39:08.060256 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:08.060219    4473 generic.go:356] "Generic (PLEG): container finished" podID="271b0cc5-8c5a-422d-b48f-85766ccc2731" containerID="4bd74d9920f6769412c06986d427f54026477900a8cd24dc5ced195dcb234b1a" exitCode=0
Nov 03 09:39:08.060648 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:08.060302    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42d4f8b1-6lr8w" event={"ID":"271b0cc5-8c5a-422d-b48f-85766ccc2731","Type":"ContainerDied","Data":"4bd74d9920f6769412c06986d427f54026477900a8cd24dc5ced195dcb234b1a"}
Nov 03 09:39:09.223566 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:09.223537    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42d4f8b1-6lr8w"
Nov 03 09:39:09.334890 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:09.334801    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/271b0cc5-8c5a-422d-b48f-85766ccc2731-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/271b0cc5-8c5a-422d-b48f-85766ccc2731-maas-api-openshift-ca\") pod \"271b0cc5-8c5a-422d-b48f-85766ccc2731\" (UID: \"271b0cc5-8c5a-422d-b48f-85766ccc2731\") "
Nov 03 09:39:09.335097 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:09.334902    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/271b0cc5-8c5a-422d-b48f-85766ccc2731-kube-api-access-zqxcb\" (UniqueName: \"kubernetes.io/projected/271b0cc5-8c5a-422d-b48f-85766ccc2731-kube-api-access-zqxcb\") pod \"271b0cc5-8c5a-422d-b48f-85766ccc2731\" (UID: \"271b0cc5-8c5a-422d-b48f-85766ccc2731\") "
Nov 03 09:39:09.335244 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:09.335098    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/271b0cc5-8c5a-422d-b48f-85766ccc2731-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/271b0cc5-8c5a-422d-b48f-85766ccc2731-maas-api-xks-ca\") pod \"271b0cc5-8c5a-422d-b48f-85766ccc2731\" (UID: \"271b0cc5-8c5a-422d-b48f-85766ccc2731\") "
Nov 03 09:39:09.335244 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:09.335155    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/271b0cc5-8c5a-422d-b48f-85766ccc2731-maas-api-openshift-ca" pod "271b0cc5-8c5a-422d-b48f-85766ccc2731" (UID: "271b0cc5-8c5a-422d-b48f-85766ccc2731"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:39:09.335514 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:09.335495    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/271b0cc5-8c5a-422d-b48f-85766ccc2731-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:39:09.337216 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:09.337195    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/271b0cc5-8c5a-422d-b48f-85766ccc2731-kube-api-access-zqxcb" pod "271b0cc5-8c5a-422d-b48f-85766ccc2731" (UID: "271b0cc5-8c5a-422d-b48f-85766ccc2731"). InnerVolumeSpecName "kube-api-access-zqxcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:39:09.337276 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:09.337261    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/271b0cc5-8c5a-422d-b48f-85766ccc2731-maas-api-xks-ca" pod "271b0cc5-8c5a-422d-b48f-85766ccc2731" (UID: "271b0cc5-8c5a-422d-b48f-85766ccc2731"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:39:09.436936 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:09.436908    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/271b0cc5-8c5a-422d-b48f-85766ccc2731-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:39:09.436936 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:09.436932    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqxcb\" (UniqueName: \"kubernetes.io/projected/271b0cc5-8c5a-422d-b48f-85766ccc2731-kube-api-access-zqxcb\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:39:10.073256 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:10.073225    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42d4f8b1-6lr8w"
Nov 03 09:39:10.073403 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:10.073257    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42d4f8b1-6lr8w" event={"ID":"271b0cc5-8c5a-422d-b48f-85766ccc2731","Type":"ContainerDied","Data":"7716399461a2b9be88e514252e282713c279fcb92e7f31d6911c96101740ce31"}
Nov 03 09:39:10.073403 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:10.073303    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7716399461a2b9be88e514252e282713c279fcb92e7f31d6911c96101740ce31"
Nov 03 09:39:16.938476 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:39:16.938440    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:39:18.958691 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:18.958661    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g"]
Nov 03 09:39:18.959216 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:18.959203    4473 cpu_manager.go:414] "containerMap: removing container" podUID="271b0cc5-8c5a-422d-b48f-85766ccc2731" containerName="revoke-keys"
Nov 03 09:39:18.959264 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:18.959217    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="271b0cc5-8c5a-422d-b48f-85766ccc2731" containerName="revoke-keys"
Nov 03 09:39:18.959335 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:18.959326    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="271b0cc5-8c5a-422d-b48f-85766ccc2731" containerName="revoke-keys"
Nov 03 09:39:18.962544 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:18.962527    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g"
Nov 03 09:39:19.017074 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:19.017049    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-05110bbd79d3\""
Nov 03 09:39:19.042576 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:19.042545    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8-provider-credentials\") pod \"payload-processing-external-model-66db7f95d6-dtd8g\" (UID: \"4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8\") " pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g"
Nov 03 09:39:19.042764 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:19.042663    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8-routing-overlay-volume\") pod \"payload-processing-external-model-66db7f95d6-dtd8g\" (UID: \"4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8\") " pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g"
Nov 03 09:39:19.042887 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:19.042856    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrr9c\" (UniqueName: \"kubernetes.io/projected/4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8-kube-api-access-rrr9c\") pod \"payload-processing-external-model-66db7f95d6-dtd8g\" (UID: \"4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8\") " pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g"
Nov 03 09:39:19.043167 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:19.043140    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8-plugins-config-volume\") pod \"payload-processing-external-model-66db7f95d6-dtd8g\" (UID: \"4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8\") " pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g"
Nov 03 09:39:19.066416 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:19.066379    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 09:39:19.066416 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:19.066404    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 09:39:19.066416 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:19.066392    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 09:39:19.066661 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:19.066446    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-gkdr4\""
Nov 03 09:39:19.066661 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:19.066535    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 09:39:19.070222 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:19.070198    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g"]
Nov 03 09:39:19.144223 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:19.144186    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrr9c\" (UniqueName: \"kubernetes.io/projected/4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8-kube-api-access-rrr9c\") pod \"payload-processing-external-model-66db7f95d6-dtd8g\" (UID: \"4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8\") " pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g"
Nov 03 09:39:19.144392 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:19.144258    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8-plugins-config-volume\") pod \"payload-processing-external-model-66db7f95d6-dtd8g\" (UID: \"4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8\") " pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g"
Nov 03 09:39:19.144392 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:19.144322    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8-provider-credentials\") pod \"payload-processing-external-model-66db7f95d6-dtd8g\" (UID: \"4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8\") " pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g"
Nov 03 09:39:19.144392 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:19.144370    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8-routing-overlay-volume\") pod \"payload-processing-external-model-66db7f95d6-dtd8g\" (UID: \"4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8\") " pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g"
Nov 03 09:39:19.145053 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:19.144995    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8-routing-overlay-volume\") pod \"payload-processing-external-model-66db7f95d6-dtd8g\" (UID: \"4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8\") " pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g"
Nov 03 09:39:19.145053 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:19.145049    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8-plugins-config-volume\") pod \"payload-processing-external-model-66db7f95d6-dtd8g\" (UID: \"4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8\") " pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g"
Nov 03 09:39:19.146893 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:19.146872    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8-provider-credentials\") pod \"payload-processing-external-model-66db7f95d6-dtd8g\" (UID: \"4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8\") " pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g"
Nov 03 09:39:19.206671 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:19.206645    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrr9c\" (UniqueName: \"kubernetes.io/projected/4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8-kube-api-access-rrr9c\") pod \"payload-processing-external-model-66db7f95d6-dtd8g\" (UID: \"4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8\") " pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g"
Nov 03 09:39:19.288598 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:19.288180    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g"
Nov 03 09:39:19.748325 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:19.748293    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g"]
Nov 03 09:39:19.749310 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:39:19.749285    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba5b3b1_ee8a_42f9_b45b_cebbfa148eb8.slice/crio-53738e0eaacc8c97a80965d922c37800953c39f936b290061de8e6f92b58db0b WatchSource:0}: Error finding container 53738e0eaacc8c97a80965d922c37800953c39f936b290061de8e6f92b58db0b: Status 404 returned error can't find the container with id 53738e0eaacc8c97a80965d922c37800953c39f936b290061de8e6f92b58db0b
Nov 03 09:39:20.133410 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:20.133376    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g" event={"ID":"4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8","Type":"ContainerStarted","Data":"f03d4e5c798d582205062c9b91c3e87f35492054ddf5557cc8aa99dce4719ef4"}
Nov 03 09:39:20.133866 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:20.133434    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g" event={"ID":"4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8","Type":"ContainerStarted","Data":"53738e0eaacc8c97a80965d922c37800953c39f936b290061de8e6f92b58db0b"}
Nov 03 09:39:20.133866 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:20.133455    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g"
Nov 03 09:39:20.252031 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:20.251980    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g" podStartSLOduration=2.251964041 podStartE2EDuration="2.251964041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:39:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:39:20.250071621 +0000 UTC m=+3584.519135056" watchObservedRunningTime="2026-11-03 09:39:20.251964041 +0000 UTC m=+3584.521027477"
Nov 03 09:39:31.154806 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:31.154761    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g" podUID="4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:39:31.284971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:31.271213    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8204d6-795bcb94cc-sbbfn"]
Nov 03 09:39:31.288967 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:31.286307    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8204d6-795bcb94cc-sbbfn"
Nov 03 09:39:31.329985 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:31.329309    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8204d6-795bcb94cc-sbbfn"]
Nov 03 09:39:31.376137 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:31.376103    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-8204d6\""
Nov 03 09:39:31.376137 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:31.376131    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-8204d6-dockercfg-95q5x\""
Nov 03 09:39:31.409311 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:31.409234    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shzdh\" (UniqueName: \"kubernetes.io/projected/f6930088-1658-49d2-a5a4-32870da3c0cc-kube-api-access-shzdh\") pod \"payload-processing-e2e-inf-a-8204d6-795bcb94cc-sbbfn\" (UID: \"f6930088-1658-49d2-a5a4-32870da3c0cc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8204d6-795bcb94cc-sbbfn"
Nov 03 09:39:31.409311 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:31.409283    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6930088-1658-49d2-a5a4-32870da3c0cc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8204d6-795bcb94cc-sbbfn\" (UID: \"f6930088-1658-49d2-a5a4-32870da3c0cc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8204d6-795bcb94cc-sbbfn"
Nov 03 09:39:31.510205 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:31.510178    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shzdh\" (UniqueName: \"kubernetes.io/projected/f6930088-1658-49d2-a5a4-32870da3c0cc-kube-api-access-shzdh\") pod \"payload-processing-e2e-inf-a-8204d6-795bcb94cc-sbbfn\" (UID: \"f6930088-1658-49d2-a5a4-32870da3c0cc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8204d6-795bcb94cc-sbbfn"
Nov 03 09:39:31.510359 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:31.510214    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6930088-1658-49d2-a5a4-32870da3c0cc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8204d6-795bcb94cc-sbbfn\" (UID: \"f6930088-1658-49d2-a5a4-32870da3c0cc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8204d6-795bcb94cc-sbbfn"
Nov 03 09:39:31.510904 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:31.510885    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6930088-1658-49d2-a5a4-32870da3c0cc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8204d6-795bcb94cc-sbbfn\" (UID: \"f6930088-1658-49d2-a5a4-32870da3c0cc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8204d6-795bcb94cc-sbbfn"
Nov 03 09:39:31.569989 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:31.569960    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shzdh\" (UniqueName: \"kubernetes.io/projected/f6930088-1658-49d2-a5a4-32870da3c0cc-kube-api-access-shzdh\") pod \"payload-processing-e2e-inf-a-8204d6-795bcb94cc-sbbfn\" (UID: \"f6930088-1658-49d2-a5a4-32870da3c0cc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8204d6-795bcb94cc-sbbfn"
Nov 03 09:39:31.624981 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:31.624934    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8204d6-795bcb94cc-sbbfn"
Nov 03 09:39:32.000474 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:32.000444    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8204d6-795bcb94cc-sbbfn"]
Nov 03 09:39:32.001755 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:39:32.001729    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6930088_1658_49d2_a5a4_32870da3c0cc.slice/crio-fdff0dbf85f7cc001399c21bfcf4a6622888b3eee5c5037199e78c6c424de7be WatchSource:0}: Error finding container fdff0dbf85f7cc001399c21bfcf4a6622888b3eee5c5037199e78c6c424de7be: Status 404 returned error can't find the container with id fdff0dbf85f7cc001399c21bfcf4a6622888b3eee5c5037199e78c6c424de7be
Nov 03 09:39:32.229624 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:32.229578    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8204d6-795bcb94cc-sbbfn" event={"ID":"f6930088-1658-49d2-a5a4-32870da3c0cc","Type":"ContainerStarted","Data":"be4dbb2678eff2ebb324fc0a8a67a719ed31d767f00bd4ea839610b068243ba2"}
Nov 03 09:39:32.230146 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:32.229676    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8204d6-795bcb94cc-sbbfn" event={"ID":"f6930088-1658-49d2-a5a4-32870da3c0cc","Type":"ContainerStarted","Data":"fdff0dbf85f7cc001399c21bfcf4a6622888b3eee5c5037199e78c6c424de7be"}
Nov 03 09:39:32.230146 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:32.229739    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8204d6-795bcb94cc-sbbfn"
Nov 03 09:39:32.347792 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:32.347696    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-8204d6-795bcb94cc-sbbfn" podStartSLOduration=1.347683056 podStartE2EDuration="1.347683056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:39:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:39:32.345916639 +0000 UTC m=+3596.614980093" watchObservedRunningTime="2026-11-03 09:39:32.347683056 +0000 UTC m=+3596.616746490"
Nov 03 09:39:34.808255 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:34.808220    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:39:36.608528 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:36.608494    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 09:39:36.609583 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:36.609561    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 09:39:41.152757 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:41.152712    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g" podUID="4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:39:43.238530 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:43.238501    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8204d6-795bcb94cc-sbbfn"
Nov 03 09:39:46.938158 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:39:46.938124    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:39:51.152368 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:39:51.152327    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g" podUID="4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:40:01.162971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:01.157727    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g" podUID="4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:40:09.917759 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:09.917721    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-dc1444-6846f74759-89k9t"]
Nov 03 09:40:09.922723 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:09.922702    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc1444-6846f74759-89k9t"
Nov 03 09:40:09.978870 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:09.978843    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-dc1444-6846f74759-89k9t"]
Nov 03 09:40:10.024787 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:10.024759    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-dc1444\""
Nov 03 09:40:10.025244 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:10.025221    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-dc1444-dockercfg-wqbtj\""
Nov 03 09:40:10.087449 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:10.087408    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bdce5e0-9b59-48dc-a4a6-851d72272b46-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-dc1444-6846f74759-89k9t\" (UID: \"3bdce5e0-9b59-48dc-a4a6-851d72272b46\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc1444-6846f74759-89k9t"
Nov 03 09:40:10.090186 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:10.090150    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn7w9\" (UniqueName: \"kubernetes.io/projected/3bdce5e0-9b59-48dc-a4a6-851d72272b46-kube-api-access-dn7w9\") pod \"payload-pre-processing-e2e-extmod-b-dc1444-6846f74759-89k9t\" (UID: \"3bdce5e0-9b59-48dc-a4a6-851d72272b46\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc1444-6846f74759-89k9t"
Nov 03 09:40:10.100229 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:10.100185    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-dc1444-67d5485fc7-l8lqz"]
Nov 03 09:40:10.127308 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:10.127279    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-dc1444-67d5485fc7-l8lqz"
Nov 03 09:40:10.146481 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:10.146451    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-dc1444-67d5485fc7-l8lqz"]
Nov 03 09:40:10.191618 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:10.191543    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bdce5e0-9b59-48dc-a4a6-851d72272b46-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-dc1444-6846f74759-89k9t\" (UID: \"3bdce5e0-9b59-48dc-a4a6-851d72272b46\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc1444-6846f74759-89k9t"
Nov 03 09:40:10.191774 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:10.191637    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn7w9\" (UniqueName: \"kubernetes.io/projected/3bdce5e0-9b59-48dc-a4a6-851d72272b46-kube-api-access-dn7w9\") pod \"payload-pre-processing-e2e-extmod-b-dc1444-6846f74759-89k9t\" (UID: \"3bdce5e0-9b59-48dc-a4a6-851d72272b46\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc1444-6846f74759-89k9t"
Nov 03 09:40:10.192215 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:10.192196    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bdce5e0-9b59-48dc-a4a6-851d72272b46-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-dc1444-6846f74759-89k9t\" (UID: \"3bdce5e0-9b59-48dc-a4a6-851d72272b46\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc1444-6846f74759-89k9t"
Nov 03 09:40:10.251306 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:10.251280    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn7w9\" (UniqueName: \"kubernetes.io/projected/3bdce5e0-9b59-48dc-a4a6-851d72272b46-kube-api-access-dn7w9\") pod \"payload-pre-processing-e2e-extmod-b-dc1444-6846f74759-89k9t\" (UID: \"3bdce5e0-9b59-48dc-a4a6-851d72272b46\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc1444-6846f74759-89k9t"
Nov 03 09:40:10.292924 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:10.292893    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fca6353-9585-4aa2-b9b0-395f9e0f0cce-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-dc1444-67d5485fc7-l8lqz\" (UID: \"2fca6353-9585-4aa2-b9b0-395f9e0f0cce\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-dc1444-67d5485fc7-l8lqz"
Nov 03 09:40:10.293096 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:10.293011    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wp2n\" (UniqueName: \"kubernetes.io/projected/2fca6353-9585-4aa2-b9b0-395f9e0f0cce-kube-api-access-9wp2n\") pod \"payload-processing-e2e-extmod-b-dc1444-67d5485fc7-l8lqz\" (UID: \"2fca6353-9585-4aa2-b9b0-395f9e0f0cce\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-dc1444-67d5485fc7-l8lqz"
Nov 03 09:40:10.393530 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:10.393498    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wp2n\" (UniqueName: \"kubernetes.io/projected/2fca6353-9585-4aa2-b9b0-395f9e0f0cce-kube-api-access-9wp2n\") pod \"payload-processing-e2e-extmod-b-dc1444-67d5485fc7-l8lqz\" (UID: \"2fca6353-9585-4aa2-b9b0-395f9e0f0cce\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-dc1444-67d5485fc7-l8lqz"
Nov 03 09:40:10.393697 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:10.393630    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fca6353-9585-4aa2-b9b0-395f9e0f0cce-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-dc1444-67d5485fc7-l8lqz\" (UID: \"2fca6353-9585-4aa2-b9b0-395f9e0f0cce\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-dc1444-67d5485fc7-l8lqz"
Nov 03 09:40:10.394366 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:10.394341    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fca6353-9585-4aa2-b9b0-395f9e0f0cce-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-dc1444-67d5485fc7-l8lqz\" (UID: \"2fca6353-9585-4aa2-b9b0-395f9e0f0cce\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-dc1444-67d5485fc7-l8lqz"
Nov 03 09:40:10.452916 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:10.452846    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wp2n\" (UniqueName: \"kubernetes.io/projected/2fca6353-9585-4aa2-b9b0-395f9e0f0cce-kube-api-access-9wp2n\") pod \"payload-processing-e2e-extmod-b-dc1444-67d5485fc7-l8lqz\" (UID: \"2fca6353-9585-4aa2-b9b0-395f9e0f0cce\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-dc1444-67d5485fc7-l8lqz"
Nov 03 09:40:10.536251 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:10.536214    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc1444-6846f74759-89k9t"
Nov 03 09:40:10.751151 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:10.751053    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-dc1444-67d5485fc7-l8lqz"
Nov 03 09:40:10.918112 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:10.918085    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-dc1444-6846f74759-89k9t"]
Nov 03 09:40:10.919823 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:40:10.919789    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bdce5e0_9b59_48dc_a4a6_851d72272b46.slice/crio-b029a1a44b6264f0503d8ae1fab78615c9f79a89570a1ef67082ad928286a1bf WatchSource:0}: Error finding container b029a1a44b6264f0503d8ae1fab78615c9f79a89570a1ef67082ad928286a1bf: Status 404 returned error can't find the container with id b029a1a44b6264f0503d8ae1fab78615c9f79a89570a1ef67082ad928286a1bf
Nov 03 09:40:11.130008 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:11.129985    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-dc1444-67d5485fc7-l8lqz"]
Nov 03 09:40:11.130531 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:40:11.130509    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fca6353_9585_4aa2_b9b0_395f9e0f0cce.slice/crio-f38f82ffa762afb438b225036e5fa834e767ed5af82adf41ef32fd9361947302 WatchSource:0}: Error finding container f38f82ffa762afb438b225036e5fa834e767ed5af82adf41ef32fd9361947302: Status 404 returned error can't find the container with id f38f82ffa762afb438b225036e5fa834e767ed5af82adf41ef32fd9361947302
Nov 03 09:40:11.153414 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:11.153379    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g" podUID="4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:40:11.493147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:11.493042    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-dc1444-67d5485fc7-l8lqz" event={"ID":"2fca6353-9585-4aa2-b9b0-395f9e0f0cce","Type":"ContainerStarted","Data":"3a3303d5606fb25bcb7d69d8c0d8e370d48ee0fd27dc91d8539c05737a1c9f5a"}
Nov 03 09:40:11.493147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:11.493103    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-dc1444-67d5485fc7-l8lqz" event={"ID":"2fca6353-9585-4aa2-b9b0-395f9e0f0cce","Type":"ContainerStarted","Data":"f38f82ffa762afb438b225036e5fa834e767ed5af82adf41ef32fd9361947302"}
Nov 03 09:40:11.493147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:11.493130    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-dc1444-67d5485fc7-l8lqz"
Nov 03 09:40:11.495014 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:11.494988    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc1444-6846f74759-89k9t" event={"ID":"3bdce5e0-9b59-48dc-a4a6-851d72272b46","Type":"ContainerStarted","Data":"98a70ddc3ddd342ff067dceb43456d566c45e6d7519c70244fe16ddba6f72da4"}
Nov 03 09:40:11.495167 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:11.495030    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc1444-6846f74759-89k9t" event={"ID":"3bdce5e0-9b59-48dc-a4a6-851d72272b46","Type":"ContainerStarted","Data":"b029a1a44b6264f0503d8ae1fab78615c9f79a89570a1ef67082ad928286a1bf"}
Nov 03 09:40:11.495167 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:11.495071    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc1444-6846f74759-89k9t"
Nov 03 09:40:11.610566 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:11.610511    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-dc1444-67d5485fc7-l8lqz" podStartSLOduration=1.610497353 podStartE2EDuration="1.610497353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:40:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:40:11.608728327 +0000 UTC m=+3635.877791762" watchObservedRunningTime="2026-11-03 09:40:11.610497353 +0000 UTC m=+3635.879560787"
Nov 03 09:40:16.938622 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:40:16.938548    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:40:21.153138 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:21.153098    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g" podUID="4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:40:22.504450 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:22.504422    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc1444-6846f74759-89k9t"
Nov 03 09:40:22.506777 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:22.504503    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-dc1444-67d5485fc7-l8lqz"
Nov 03 09:40:22.622829 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:22.622769    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc1444-6846f74759-89k9t" podStartSLOduration=13.62275045 podStartE2EDuration="13.62275045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:40:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:40:11.749255377 +0000 UTC m=+3636.018319108" watchObservedRunningTime="2026-11-03 09:40:22.62275045 +0000 UTC m=+3646.891813887"
Nov 03 09:40:26.794022 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:26.793984    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:40:31.152909 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:31.152870    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g" podUID="4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:40:37.902708 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:37.902662    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-af514d-854f5df6d5-7xbnz"]
Nov 03 09:40:37.910330 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:37.910300    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-af514d-854f5df6d5-7xbnz"
Nov 03 09:40:37.916393 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:37.916363    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-af514d-854f5df6d5-7xbnz"]
Nov 03 09:40:37.965131 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:37.965104    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-af514d\""
Nov 03 09:40:37.994829 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:37.994799    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wkl4\" (UniqueName: \"kubernetes.io/projected/94aabaf0-2118-41f5-99b8-e558c95a5f46-kube-api-access-5wkl4\") pod \"payload-processing-e2e-inf-b-af514d-854f5df6d5-7xbnz\" (UID: \"94aabaf0-2118-41f5-99b8-e558c95a5f46\") " pod="openshift-ingress/payload-processing-e2e-inf-b-af514d-854f5df6d5-7xbnz"
Nov 03 09:40:37.994992 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:37.994832    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94aabaf0-2118-41f5-99b8-e558c95a5f46-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-af514d-854f5df6d5-7xbnz\" (UID: \"94aabaf0-2118-41f5-99b8-e558c95a5f46\") " pod="openshift-ingress/payload-processing-e2e-inf-b-af514d-854f5df6d5-7xbnz"
Nov 03 09:40:38.014202 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:38.014183    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-af514d-dockercfg-rkg87\""
Nov 03 09:40:38.096539 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:38.096501    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wkl4\" (UniqueName: \"kubernetes.io/projected/94aabaf0-2118-41f5-99b8-e558c95a5f46-kube-api-access-5wkl4\") pod \"payload-processing-e2e-inf-b-af514d-854f5df6d5-7xbnz\" (UID: \"94aabaf0-2118-41f5-99b8-e558c95a5f46\") " pod="openshift-ingress/payload-processing-e2e-inf-b-af514d-854f5df6d5-7xbnz"
Nov 03 09:40:38.096539 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:38.096537    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94aabaf0-2118-41f5-99b8-e558c95a5f46-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-af514d-854f5df6d5-7xbnz\" (UID: \"94aabaf0-2118-41f5-99b8-e558c95a5f46\") " pod="openshift-ingress/payload-processing-e2e-inf-b-af514d-854f5df6d5-7xbnz"
Nov 03 09:40:38.097287 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:38.097266    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94aabaf0-2118-41f5-99b8-e558c95a5f46-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-af514d-854f5df6d5-7xbnz\" (UID: \"94aabaf0-2118-41f5-99b8-e558c95a5f46\") " pod="openshift-ingress/payload-processing-e2e-inf-b-af514d-854f5df6d5-7xbnz"
Nov 03 09:40:38.155586 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:38.155524    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wkl4\" (UniqueName: \"kubernetes.io/projected/94aabaf0-2118-41f5-99b8-e558c95a5f46-kube-api-access-5wkl4\") pod \"payload-processing-e2e-inf-b-af514d-854f5df6d5-7xbnz\" (UID: \"94aabaf0-2118-41f5-99b8-e558c95a5f46\") " pod="openshift-ingress/payload-processing-e2e-inf-b-af514d-854f5df6d5-7xbnz"
Nov 03 09:40:38.231254 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:38.231230    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-af514d-854f5df6d5-7xbnz"
Nov 03 09:40:38.607643 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:38.607618    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-af514d-854f5df6d5-7xbnz"]
Nov 03 09:40:38.608542 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:40:38.608516    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94aabaf0_2118_41f5_99b8_e558c95a5f46.slice/crio-b93dba53bce225fbfefe2a7b66535f438260dfd70e9e4c9d54a678d50a9d9f5e WatchSource:0}: Error finding container b93dba53bce225fbfefe2a7b66535f438260dfd70e9e4c9d54a678d50a9d9f5e: Status 404 returned error can't find the container with id b93dba53bce225fbfefe2a7b66535f438260dfd70e9e4c9d54a678d50a9d9f5e
Nov 03 09:40:38.682023 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:38.681995    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-af514d-854f5df6d5-7xbnz" event={"ID":"94aabaf0-2118-41f5-99b8-e558c95a5f46","Type":"ContainerStarted","Data":"0894daf4cc73ae78455c4c59a88538cc94662404472a7043d0b1e19314992915"}
Nov 03 09:40:38.682141 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:38.682056    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-af514d-854f5df6d5-7xbnz" event={"ID":"94aabaf0-2118-41f5-99b8-e558c95a5f46","Type":"ContainerStarted","Data":"b93dba53bce225fbfefe2a7b66535f438260dfd70e9e4c9d54a678d50a9d9f5e"}
Nov 03 09:40:38.682141 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:38.682102    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-af514d-854f5df6d5-7xbnz"
Nov 03 09:40:38.802477 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:38.802417    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-af514d-854f5df6d5-7xbnz" podStartSLOduration=1.802398831 podStartE2EDuration="1.802398831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:40:38.799749056 +0000 UTC m=+3663.068812538" watchObservedRunningTime="2026-11-03 09:40:38.802398831 +0000 UTC m=+3663.071462266"
Nov 03 09:40:39.864191 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:39.864149    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g" podUID="4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:40:46.937647 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:40:46.937610    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:40:49.700693 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:49.700654    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-af514d-854f5df6d5-7xbnz"
Nov 03 09:40:49.864339 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:49.864299    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g" podUID="4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:40:59.864889 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:40:59.864844    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g" podUID="4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:41:09.865010 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:09.864974    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g" podUID="4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:41:16.940040 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:41:16.940006    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:41:19.751983 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:19.751925    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:41:19.864334 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:19.864301    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g" podUID="4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:41:29.864812 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:29.864775    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g" podUID="4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:41:30.042023 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:30.041985    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ff6461-6884958899-5sml8"]
Nov 03 09:41:30.255816 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:30.252658    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ff6461-6ccb9d7b95-kklnx"]
Nov 03 09:41:35.042963 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:35.042890    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-ff6461-6884958899-5sml8" podUID="0c195d47-54ec-435a-9a6d-9f5d5caa5131" containerName="payload-processing" containerID="cri-o://2591a2b76ecd6c6a3ea50c001b64a7603035b2dd5d285befdbf5e71a373ab884" gracePeriod=25
Nov 03 09:41:35.253722 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:35.253664    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff6461-6ccb9d7b95-kklnx" podUID="a02abcb7-266e-40cd-a1fb-fca2b5706f14" containerName="payload-pre-processing" containerID="cri-o://066fb199953d3a91f831957ce59087fd3d47829e510e2e915bd1e10378675601" gracePeriod=25
Nov 03 09:41:35.385843 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:35.385817    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ff6461-6884958899-5sml8"
Nov 03 09:41:35.491126 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:35.491103    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c195d47-54ec-435a-9a6d-9f5d5caa5131-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c195d47-54ec-435a-9a6d-9f5d5caa5131-plugins-config-volume\") pod \"0c195d47-54ec-435a-9a6d-9f5d5caa5131\" (UID: \"0c195d47-54ec-435a-9a6d-9f5d5caa5131\") "
Nov 03 09:41:35.491268 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:35.491230    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c195d47-54ec-435a-9a6d-9f5d5caa5131-kube-api-access-j485n\" (UniqueName: \"kubernetes.io/projected/0c195d47-54ec-435a-9a6d-9f5d5caa5131-kube-api-access-j485n\") pod \"0c195d47-54ec-435a-9a6d-9f5d5caa5131\" (UID: \"0c195d47-54ec-435a-9a6d-9f5d5caa5131\") "
Nov 03 09:41:35.491508 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:35.491468    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c195d47-54ec-435a-9a6d-9f5d5caa5131-plugins-config-volume" pod "0c195d47-54ec-435a-9a6d-9f5d5caa5131" (UID: "0c195d47-54ec-435a-9a6d-9f5d5caa5131"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:41:35.491763 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:35.491747    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c195d47-54ec-435a-9a6d-9f5d5caa5131-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:41:35.493401 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:35.493377    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c195d47-54ec-435a-9a6d-9f5d5caa5131-kube-api-access-j485n" pod "0c195d47-54ec-435a-9a6d-9f5d5caa5131" (UID: "0c195d47-54ec-435a-9a6d-9f5d5caa5131"). InnerVolumeSpecName "kube-api-access-j485n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:41:35.499621 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:35.499600    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff6461-6ccb9d7b95-kklnx"
Nov 03 09:41:35.593168 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:35.593078    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a02abcb7-266e-40cd-a1fb-fca2b5706f14-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a02abcb7-266e-40cd-a1fb-fca2b5706f14-plugins-config-volume\") pod \"a02abcb7-266e-40cd-a1fb-fca2b5706f14\" (UID: \"a02abcb7-266e-40cd-a1fb-fca2b5706f14\") "
Nov 03 09:41:35.593168 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:35.593141    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a02abcb7-266e-40cd-a1fb-fca2b5706f14-kube-api-access-7ptsb\" (UniqueName: \"kubernetes.io/projected/a02abcb7-266e-40cd-a1fb-fca2b5706f14-kube-api-access-7ptsb\") pod \"a02abcb7-266e-40cd-a1fb-fca2b5706f14\" (UID: \"a02abcb7-266e-40cd-a1fb-fca2b5706f14\") "
Nov 03 09:41:35.593484 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:35.593471    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j485n\" (UniqueName: \"kubernetes.io/projected/0c195d47-54ec-435a-9a6d-9f5d5caa5131-kube-api-access-j485n\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:41:35.593548 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:35.593497    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a02abcb7-266e-40cd-a1fb-fca2b5706f14-plugins-config-volume" pod "a02abcb7-266e-40cd-a1fb-fca2b5706f14" (UID: "a02abcb7-266e-40cd-a1fb-fca2b5706f14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:41:35.595403 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:35.595384    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a02abcb7-266e-40cd-a1fb-fca2b5706f14-kube-api-access-7ptsb" pod "a02abcb7-266e-40cd-a1fb-fca2b5706f14" (UID: "a02abcb7-266e-40cd-a1fb-fca2b5706f14"). InnerVolumeSpecName "kube-api-access-7ptsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:41:35.694977 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:35.694928    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a02abcb7-266e-40cd-a1fb-fca2b5706f14-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:41:35.694977 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:35.694971    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ptsb\" (UniqueName: \"kubernetes.io/projected/a02abcb7-266e-40cd-a1fb-fca2b5706f14-kube-api-access-7ptsb\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:41:36.053285 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:36.053249    4473 generic.go:356] "Generic (PLEG): container finished" podID="0c195d47-54ec-435a-9a6d-9f5d5caa5131" containerID="2591a2b76ecd6c6a3ea50c001b64a7603035b2dd5d285befdbf5e71a373ab884" exitCode=0
Nov 03 09:41:36.053737 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:36.053321    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ff6461-6884958899-5sml8"
Nov 03 09:41:36.053737 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:36.053335    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ff6461-6884958899-5sml8" event={"ID":"0c195d47-54ec-435a-9a6d-9f5d5caa5131","Type":"ContainerDied","Data":"2591a2b76ecd6c6a3ea50c001b64a7603035b2dd5d285befdbf5e71a373ab884"}
Nov 03 09:41:36.053737 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:36.053392    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ff6461-6884958899-5sml8" event={"ID":"0c195d47-54ec-435a-9a6d-9f5d5caa5131","Type":"ContainerDied","Data":"35edfff3597010684b3a61be8ccc17115f9b18ff625f48d54128d119fddd8100"}
Nov 03 09:41:36.053737 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:36.053420    4473 scope.go:122] "RemoveContainer" containerID="2591a2b76ecd6c6a3ea50c001b64a7603035b2dd5d285befdbf5e71a373ab884"
Nov 03 09:41:36.054913 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:36.054884    4473 generic.go:356] "Generic (PLEG): container finished" podID="a02abcb7-266e-40cd-a1fb-fca2b5706f14" containerID="066fb199953d3a91f831957ce59087fd3d47829e510e2e915bd1e10378675601" exitCode=0
Nov 03 09:41:36.055017 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:36.054989    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff6461-6ccb9d7b95-kklnx"
Nov 03 09:41:36.055017 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:36.055003    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff6461-6ccb9d7b95-kklnx" event={"ID":"a02abcb7-266e-40cd-a1fb-fca2b5706f14","Type":"ContainerDied","Data":"066fb199953d3a91f831957ce59087fd3d47829e510e2e915bd1e10378675601"}
Nov 03 09:41:36.055132 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:36.055039    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ff6461-6ccb9d7b95-kklnx" event={"ID":"a02abcb7-266e-40cd-a1fb-fca2b5706f14","Type":"ContainerDied","Data":"a2b658990c0ac67a27f75aaaaa1afe968b43a0f95781f4e427a3df7b7106b9ec"}
Nov 03 09:41:36.075866 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:36.075845    4473 scope.go:122] "RemoveContainer" containerID="2591a2b76ecd6c6a3ea50c001b64a7603035b2dd5d285befdbf5e71a373ab884"
Nov 03 09:41:36.076212 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:41:36.076188    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2591a2b76ecd6c6a3ea50c001b64a7603035b2dd5d285befdbf5e71a373ab884\": container with ID starting with 2591a2b76ecd6c6a3ea50c001b64a7603035b2dd5d285befdbf5e71a373ab884 not found: ID does not exist" containerID="2591a2b76ecd6c6a3ea50c001b64a7603035b2dd5d285befdbf5e71a373ab884"
Nov 03 09:41:36.076308 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:36.076220    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2591a2b76ecd6c6a3ea50c001b64a7603035b2dd5d285befdbf5e71a373ab884"} err="failed to get container status \"2591a2b76ecd6c6a3ea50c001b64a7603035b2dd5d285befdbf5e71a373ab884\": rpc error: code = NotFound desc = could not find container \"2591a2b76ecd6c6a3ea50c001b64a7603035b2dd5d285befdbf5e71a373ab884\": container with ID starting with 2591a2b76ecd6c6a3ea50c001b64a7603035b2dd5d285befdbf5e71a373ab884 not found: ID does not exist"
Nov 03 09:41:36.076308 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:36.076239    4473 scope.go:122] "RemoveContainer" containerID="066fb199953d3a91f831957ce59087fd3d47829e510e2e915bd1e10378675601"
Nov 03 09:41:36.085161 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:36.085140    4473 scope.go:122] "RemoveContainer" containerID="066fb199953d3a91f831957ce59087fd3d47829e510e2e915bd1e10378675601"
Nov 03 09:41:36.085431 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:41:36.085405    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"066fb199953d3a91f831957ce59087fd3d47829e510e2e915bd1e10378675601\": container with ID starting with 066fb199953d3a91f831957ce59087fd3d47829e510e2e915bd1e10378675601 not found: ID does not exist" containerID="066fb199953d3a91f831957ce59087fd3d47829e510e2e915bd1e10378675601"
Nov 03 09:41:36.085507 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:36.085433    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"066fb199953d3a91f831957ce59087fd3d47829e510e2e915bd1e10378675601"} err="failed to get container status \"066fb199953d3a91f831957ce59087fd3d47829e510e2e915bd1e10378675601\": rpc error: code = NotFound desc = could not find container \"066fb199953d3a91f831957ce59087fd3d47829e510e2e915bd1e10378675601\": container with ID starting with 066fb199953d3a91f831957ce59087fd3d47829e510e2e915bd1e10378675601 not found: ID does not exist"
Nov 03 09:41:36.222762 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:36.222683    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ff6461-6ccb9d7b95-kklnx"]
Nov 03 09:41:36.230689 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:36.230664    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ff6461-6ccb9d7b95-kklnx"]
Nov 03 09:41:36.345658 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:36.345625    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ff6461-6884958899-5sml8"]
Nov 03 09:41:36.354330 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:36.354299    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ff6461-6884958899-5sml8"]
Nov 03 09:41:37.870861 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:37.870818    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c195d47-54ec-435a-9a6d-9f5d5caa5131" path="/var/lib/kubelet/pods/0c195d47-54ec-435a-9a6d-9f5d5caa5131/volumes"
Nov 03 09:41:37.871452 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:37.871431    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a02abcb7-266e-40cd-a1fb-fca2b5706f14" path="/var/lib/kubelet/pods/a02abcb7-266e-40cd-a1fb-fca2b5706f14/volumes"
Nov 03 09:41:38.599661 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:38.599632    4473 scope.go:122] "RemoveContainer" containerID="35178f4984668950c4f9944959121146e89e44c578cf3513024941e6e1cee1c4"
Nov 03 09:41:38.609565 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:38.609542    4473 scope.go:122] "RemoveContainer" containerID="68b9f4b768e9034410a996bbc36ea2919b6c206533dc44f49245ae763b9373dd"
Nov 03 09:41:39.865097 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:39.865052    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g" podUID="4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:41:46.937510 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:41:46.937428    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:41:49.864536 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:49.864474    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g" podUID="4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:41:59.864357 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:41:59.864322    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g" podUID="4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:42:02.569273 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:42:02.569235    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:42:02.864367 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:42:02.864281    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g" podUID="4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:42:05.052583 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:42:05.052550    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8af98dc3-2hsmf"]
Nov 03 09:42:05.102765 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:42:05.102738    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8af98dc3-2hsmf"]
Nov 03 09:42:05.870203 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:42:05.870167    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06434827-ae83-450d-8afa-104d54b147b6" path="/var/lib/kubelet/pods/06434827-ae83-450d-8afa-104d54b147b6/volumes"
Nov 03 09:42:12.864300 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:42:12.864259    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g" podUID="4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:42:16.937658 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:42:16.937625    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:42:22.864146 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:42:22.864114    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g" podUID="4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:42:32.864263 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:42:32.864230    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g" podUID="4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:42:34.603021 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:42:34.602992    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8204d6-795bcb94cc-sbbfn"]
Nov 03 09:42:37.153764 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:42:37.153722    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:42:39.603881 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:42:39.603836    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-8204d6-795bcb94cc-sbbfn" podUID="f6930088-1658-49d2-a5a4-32870da3c0cc" containerName="payload-processing" containerID="cri-o://be4dbb2678eff2ebb324fc0a8a67a719ed31d767f00bd4ea839610b068243ba2" gracePeriod=25
Nov 03 09:42:39.859036 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:42:39.858980    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8204d6-795bcb94cc-sbbfn"
Nov 03 09:42:39.907099 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:42:39.907070    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6930088-1658-49d2-a5a4-32870da3c0cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6930088-1658-49d2-a5a4-32870da3c0cc-plugins-config-volume\") pod \"f6930088-1658-49d2-a5a4-32870da3c0cc\" (UID: \"f6930088-1658-49d2-a5a4-32870da3c0cc\") "
Nov 03 09:42:39.907265 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:42:39.907195    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6930088-1658-49d2-a5a4-32870da3c0cc-kube-api-access-shzdh\" (UniqueName: \"kubernetes.io/projected/f6930088-1658-49d2-a5a4-32870da3c0cc-kube-api-access-shzdh\") pod \"f6930088-1658-49d2-a5a4-32870da3c0cc\" (UID: \"f6930088-1658-49d2-a5a4-32870da3c0cc\") "
Nov 03 09:42:39.907438 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:42:39.907416    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6930088-1658-49d2-a5a4-32870da3c0cc-plugins-config-volume" pod "f6930088-1658-49d2-a5a4-32870da3c0cc" (UID: "f6930088-1658-49d2-a5a4-32870da3c0cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:42:39.907624 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:42:39.907607    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6930088-1658-49d2-a5a4-32870da3c0cc-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:42:39.909480 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:42:39.909459    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6930088-1658-49d2-a5a4-32870da3c0cc-kube-api-access-shzdh" pod "f6930088-1658-49d2-a5a4-32870da3c0cc" (UID: "f6930088-1658-49d2-a5a4-32870da3c0cc"). InnerVolumeSpecName "kube-api-access-shzdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:42:40.009144 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:42:40.009107    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shzdh\" (UniqueName: \"kubernetes.io/projected/f6930088-1658-49d2-a5a4-32870da3c0cc-kube-api-access-shzdh\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:42:40.526213 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:42:40.526175    4473 generic.go:356] "Generic (PLEG): container finished" podID="f6930088-1658-49d2-a5a4-32870da3c0cc" containerID="be4dbb2678eff2ebb324fc0a8a67a719ed31d767f00bd4ea839610b068243ba2" exitCode=0
Nov 03 09:42:40.526403 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:42:40.526227    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8204d6-795bcb94cc-sbbfn"
Nov 03 09:42:40.526403 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:42:40.526231    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8204d6-795bcb94cc-sbbfn" event={"ID":"f6930088-1658-49d2-a5a4-32870da3c0cc","Type":"ContainerDied","Data":"be4dbb2678eff2ebb324fc0a8a67a719ed31d767f00bd4ea839610b068243ba2"}
Nov 03 09:42:40.526403 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:42:40.526283    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8204d6-795bcb94cc-sbbfn" event={"ID":"f6930088-1658-49d2-a5a4-32870da3c0cc","Type":"ContainerDied","Data":"fdff0dbf85f7cc001399c21bfcf4a6622888b3eee5c5037199e78c6c424de7be"}
Nov 03 09:42:40.526403 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:42:40.526313    4473 scope.go:122] "RemoveContainer" containerID="be4dbb2678eff2ebb324fc0a8a67a719ed31d767f00bd4ea839610b068243ba2"
Nov 03 09:42:40.539609 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:42:40.539585    4473 scope.go:122] "RemoveContainer" containerID="be4dbb2678eff2ebb324fc0a8a67a719ed31d767f00bd4ea839610b068243ba2"
Nov 03 09:42:40.539915 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:42:40.539895    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be4dbb2678eff2ebb324fc0a8a67a719ed31d767f00bd4ea839610b068243ba2\": container with ID starting with be4dbb2678eff2ebb324fc0a8a67a719ed31d767f00bd4ea839610b068243ba2 not found: ID does not exist" containerID="be4dbb2678eff2ebb324fc0a8a67a719ed31d767f00bd4ea839610b068243ba2"
Nov 03 09:42:40.540004 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:42:40.539924    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be4dbb2678eff2ebb324fc0a8a67a719ed31d767f00bd4ea839610b068243ba2"} err="failed to get container status \"be4dbb2678eff2ebb324fc0a8a67a719ed31d767f00bd4ea839610b068243ba2\": rpc error: code = NotFound desc = could not find container \"be4dbb2678eff2ebb324fc0a8a67a719ed31d767f00bd4ea839610b068243ba2\": container with ID starting with be4dbb2678eff2ebb324fc0a8a67a719ed31d767f00bd4ea839610b068243ba2 not found: ID does not exist"
Nov 03 09:42:40.697593 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:42:40.697559    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8204d6-795bcb94cc-sbbfn"]
Nov 03 09:42:40.748080 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:42:40.748039    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8204d6-795bcb94cc-sbbfn"]
Nov 03 09:42:41.871421 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:42:41.871389    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6930088-1658-49d2-a5a4-32870da3c0cc" path="/var/lib/kubelet/pods/f6930088-1658-49d2-a5a4-32870da3c0cc/volumes"
Nov 03 09:42:42.864104 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:42:42.864072    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g" podUID="4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:42:46.938182 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:42:46.938149    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:42:52.868399 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:42:52.868363    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g" podUID="4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:43:02.864335 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:02.864296    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g" podUID="4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:43:12.864852 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:12.864816    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g" podUID="4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:43:15.385859 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:15.385826    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-373116-5645958765-96jxp"]
Nov 03 09:43:15.499725 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:15.499608    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-373116-759d64f4b7-r4dmq"]
Nov 03 09:43:16.955794 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:43:16.955703    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:43:18.344552 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:18.344501    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:43:20.390052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:20.390001    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-373116-5645958765-96jxp" podUID="2ff5c62e-0eaa-45d9-86db-8636677353fc" containerName="payload-processing" containerID="cri-o://3eea2681da7998aa52762905937334aa65bbb172f7c5bf4aa7ee5392e39521dc" gracePeriod=25
Nov 03 09:43:20.506971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:20.506003    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-373116-759d64f4b7-r4dmq" podUID="d6f5b134-d645-43dc-a388-10e05dc52168" containerName="payload-pre-processing" containerID="cri-o://67abebd98d7a9bc3dd69f963825de232ddc90b1f84f8691bbc6ce4945924269b" gracePeriod=25
Nov 03 09:43:21.002110 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:21.001625    4473 generic.go:356] "Generic (PLEG): container finished" podID="2ff5c62e-0eaa-45d9-86db-8636677353fc" containerID="3eea2681da7998aa52762905937334aa65bbb172f7c5bf4aa7ee5392e39521dc" exitCode=0
Nov 03 09:43:21.002110 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:21.001727    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-373116-5645958765-96jxp" event={"ID":"2ff5c62e-0eaa-45d9-86db-8636677353fc","Type":"ContainerDied","Data":"3eea2681da7998aa52762905937334aa65bbb172f7c5bf4aa7ee5392e39521dc"}
Nov 03 09:43:21.002110 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:21.001783    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-373116-5645958765-96jxp" event={"ID":"2ff5c62e-0eaa-45d9-86db-8636677353fc","Type":"ContainerDied","Data":"4bccfcd96198eedbc15045c925b19d081939560c6ce198d9ffc25b8cd9dbe823"}
Nov 03 09:43:21.002420 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:21.002135    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bccfcd96198eedbc15045c925b19d081939560c6ce198d9ffc25b8cd9dbe823"
Nov 03 09:43:21.004503 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:21.004477    4473 generic.go:356] "Generic (PLEG): container finished" podID="d6f5b134-d645-43dc-a388-10e05dc52168" containerID="67abebd98d7a9bc3dd69f963825de232ddc90b1f84f8691bbc6ce4945924269b" exitCode=0
Nov 03 09:43:21.004851 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:21.004829    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-373116-759d64f4b7-r4dmq" event={"ID":"d6f5b134-d645-43dc-a388-10e05dc52168","Type":"ContainerDied","Data":"67abebd98d7a9bc3dd69f963825de232ddc90b1f84f8691bbc6ce4945924269b"}
Nov 03 09:43:21.110515 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:21.110487    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-373116-5645958765-96jxp"
Nov 03 09:43:21.212087 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:21.211157    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ff5c62e-0eaa-45d9-86db-8636677353fc-kube-api-access-6bxws\" (UniqueName: \"kubernetes.io/projected/2ff5c62e-0eaa-45d9-86db-8636677353fc-kube-api-access-6bxws\") pod \"2ff5c62e-0eaa-45d9-86db-8636677353fc\" (UID: \"2ff5c62e-0eaa-45d9-86db-8636677353fc\") "
Nov 03 09:43:21.212087 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:21.211332    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ff5c62e-0eaa-45d9-86db-8636677353fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ff5c62e-0eaa-45d9-86db-8636677353fc-plugins-config-volume\") pod \"2ff5c62e-0eaa-45d9-86db-8636677353fc\" (UID: \"2ff5c62e-0eaa-45d9-86db-8636677353fc\") "
Nov 03 09:43:21.212876 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:21.212535    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ff5c62e-0eaa-45d9-86db-8636677353fc-plugins-config-volume" pod "2ff5c62e-0eaa-45d9-86db-8636677353fc" (UID: "2ff5c62e-0eaa-45d9-86db-8636677353fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:43:21.297044 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:21.258087    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff5c62e-0eaa-45d9-86db-8636677353fc-kube-api-access-6bxws" pod "2ff5c62e-0eaa-45d9-86db-8636677353fc" (UID: "2ff5c62e-0eaa-45d9-86db-8636677353fc"). InnerVolumeSpecName "kube-api-access-6bxws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:43:21.322073 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:21.313818    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bxws\" (UniqueName: \"kubernetes.io/projected/2ff5c62e-0eaa-45d9-86db-8636677353fc-kube-api-access-6bxws\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:43:21.322303 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:21.322280    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ff5c62e-0eaa-45d9-86db-8636677353fc-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:43:21.405384 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:21.405351    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-373116-759d64f4b7-r4dmq"
Nov 03 09:43:21.436970 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:21.436308    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6f5b134-d645-43dc-a388-10e05dc52168-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f5b134-d645-43dc-a388-10e05dc52168-plugins-config-volume\") pod \"d6f5b134-d645-43dc-a388-10e05dc52168\" (UID: \"d6f5b134-d645-43dc-a388-10e05dc52168\") "
Nov 03 09:43:21.436970 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:21.436474    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6f5b134-d645-43dc-a388-10e05dc52168-kube-api-access-q7l62\" (UniqueName: \"kubernetes.io/projected/d6f5b134-d645-43dc-a388-10e05dc52168-kube-api-access-q7l62\") pod \"d6f5b134-d645-43dc-a388-10e05dc52168\" (UID: \"d6f5b134-d645-43dc-a388-10e05dc52168\") "
Nov 03 09:43:21.497203 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:21.495090    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6f5b134-d645-43dc-a388-10e05dc52168-plugins-config-volume" pod "d6f5b134-d645-43dc-a388-10e05dc52168" (UID: "d6f5b134-d645-43dc-a388-10e05dc52168"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:43:21.499228 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:21.499111    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6f5b134-d645-43dc-a388-10e05dc52168-kube-api-access-q7l62" pod "d6f5b134-d645-43dc-a388-10e05dc52168" (UID: "d6f5b134-d645-43dc-a388-10e05dc52168"). InnerVolumeSpecName "kube-api-access-q7l62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:43:21.545389 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:21.545326    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f5b134-d645-43dc-a388-10e05dc52168-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:43:21.545389 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:21.545362    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7l62\" (UniqueName: \"kubernetes.io/projected/d6f5b134-d645-43dc-a388-10e05dc52168-kube-api-access-q7l62\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:43:22.013292 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:22.013260    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-373116-759d64f4b7-r4dmq"
Nov 03 09:43:22.013461 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:22.013289    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-373116-759d64f4b7-r4dmq" event={"ID":"d6f5b134-d645-43dc-a388-10e05dc52168","Type":"ContainerDied","Data":"4ce10b2c0b25351c521d410e6053fa79f79fac30dbe42468d80f8c66d1e5f759"}
Nov 03 09:43:22.013461 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:22.013368    4473 scope.go:122] "RemoveContainer" containerID="67abebd98d7a9bc3dd69f963825de232ddc90b1f84f8691bbc6ce4945924269b"
Nov 03 09:43:22.013739 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:22.013719    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-373116-5645958765-96jxp"
Nov 03 09:43:22.188595 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:22.188502    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-373116-759d64f4b7-r4dmq"]
Nov 03 09:43:22.235185 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:22.235154    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-373116-759d64f4b7-r4dmq"]
Nov 03 09:43:22.302769 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:22.302739    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-373116-5645958765-96jxp"]
Nov 03 09:43:22.307922 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:22.307898    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-373116-5645958765-96jxp"]
Nov 03 09:43:22.864741 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:22.864702    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g" podUID="4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:43:23.870261 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:23.870222    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ff5c62e-0eaa-45d9-86db-8636677353fc" path="/var/lib/kubelet/pods/2ff5c62e-0eaa-45d9-86db-8636677353fc/volumes"
Nov 03 09:43:23.870780 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:23.870758    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6f5b134-d645-43dc-a388-10e05dc52168" path="/var/lib/kubelet/pods/d6f5b134-d645-43dc-a388-10e05dc52168/volumes"
Nov 03 09:43:29.864337 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:29.864300    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g" podUID="4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:43:38.655807 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:38.655776    4473 scope.go:122] "RemoveContainer" containerID="0ed7e42671dccdf2c188e8d1e8b79003081d6dd5f888d23754b5bd0efa5953b0"
Nov 03 09:43:38.665508 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:38.665488    4473 scope.go:122] "RemoveContainer" containerID="3eea2681da7998aa52762905937334aa65bbb172f7c5bf4aa7ee5392e39521dc"
Nov 03 09:43:39.865156 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:39.865113    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g" podUID="4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:43:44.473867 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:44.473831    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-872281-5d9f945fc-wnl4l"]
Nov 03 09:43:44.580456 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:44.580424    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-872281-546f59c95-7tpjd"]
Nov 03 09:43:46.937327 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:43:46.937293    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:43:49.474467 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:49.474407    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-872281-5d9f945fc-wnl4l" podUID="c4e5a6e1-68ac-4090-805f-e0a944f505b9" containerName="payload-processing" containerID="cri-o://8d93dd4061a6eb49b43563f03a4d294ece33590d5356fdd122c70156da14da23" gracePeriod=25
Nov 03 09:43:49.580888 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:49.580856    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872281-546f59c95-7tpjd" podUID="341adb1c-5c4f-41ce-a0e4-d929d8b32121" containerName="payload-pre-processing" containerID="cri-o://33463b1eb9bc5179227ce4525a53f2b38ab431eb09fe2d60c57dfce5eac627f1" gracePeriod=25
Nov 03 09:43:49.745645 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:49.745575    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-872281-5d9f945fc-wnl4l"
Nov 03 09:43:49.804237 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:49.804207    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4e5a6e1-68ac-4090-805f-e0a944f505b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e5a6e1-68ac-4090-805f-e0a944f505b9-plugins-config-volume\") pod \"c4e5a6e1-68ac-4090-805f-e0a944f505b9\" (UID: \"c4e5a6e1-68ac-4090-805f-e0a944f505b9\") "
Nov 03 09:43:49.804476 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:49.804386    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4e5a6e1-68ac-4090-805f-e0a944f505b9-kube-api-access-78sk4\" (UniqueName: \"kubernetes.io/projected/c4e5a6e1-68ac-4090-805f-e0a944f505b9-kube-api-access-78sk4\") pod \"c4e5a6e1-68ac-4090-805f-e0a944f505b9\" (UID: \"c4e5a6e1-68ac-4090-805f-e0a944f505b9\") "
Nov 03 09:43:49.804572 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:49.804535    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4e5a6e1-68ac-4090-805f-e0a944f505b9-plugins-config-volume" pod "c4e5a6e1-68ac-4090-805f-e0a944f505b9" (UID: "c4e5a6e1-68ac-4090-805f-e0a944f505b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:43:49.804765 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:49.804745    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e5a6e1-68ac-4090-805f-e0a944f505b9-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:43:49.807697 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:49.807670    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4e5a6e1-68ac-4090-805f-e0a944f505b9-kube-api-access-78sk4" pod "c4e5a6e1-68ac-4090-805f-e0a944f505b9" (UID: "c4e5a6e1-68ac-4090-805f-e0a944f505b9"). InnerVolumeSpecName "kube-api-access-78sk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:43:49.842609 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:49.842588    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872281-546f59c95-7tpjd"
Nov 03 09:43:49.864608 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:49.864576    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g" podUID="4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:43:49.905551 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:49.905522    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/341adb1c-5c4f-41ce-a0e4-d929d8b32121-kube-api-access-d9s42\" (UniqueName: \"kubernetes.io/projected/341adb1c-5c4f-41ce-a0e4-d929d8b32121-kube-api-access-d9s42\") pod \"341adb1c-5c4f-41ce-a0e4-d929d8b32121\" (UID: \"341adb1c-5c4f-41ce-a0e4-d929d8b32121\") "
Nov 03 09:43:49.905708 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:49.905612    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/341adb1c-5c4f-41ce-a0e4-d929d8b32121-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/341adb1c-5c4f-41ce-a0e4-d929d8b32121-plugins-config-volume\") pod \"341adb1c-5c4f-41ce-a0e4-d929d8b32121\" (UID: \"341adb1c-5c4f-41ce-a0e4-d929d8b32121\") "
Nov 03 09:43:49.906066 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:49.906034    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/341adb1c-5c4f-41ce-a0e4-d929d8b32121-plugins-config-volume" pod "341adb1c-5c4f-41ce-a0e4-d929d8b32121" (UID: "341adb1c-5c4f-41ce-a0e4-d929d8b32121"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:43:49.906219 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:49.906185    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78sk4\" (UniqueName: \"kubernetes.io/projected/c4e5a6e1-68ac-4090-805f-e0a944f505b9-kube-api-access-78sk4\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:43:49.906219 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:49.906209    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/341adb1c-5c4f-41ce-a0e4-d929d8b32121-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:43:49.907837 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:49.907810    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/341adb1c-5c4f-41ce-a0e4-d929d8b32121-kube-api-access-d9s42" pod "341adb1c-5c4f-41ce-a0e4-d929d8b32121" (UID: "341adb1c-5c4f-41ce-a0e4-d929d8b32121"). InnerVolumeSpecName "kube-api-access-d9s42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:43:50.007292 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:50.007219    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9s42\" (UniqueName: \"kubernetes.io/projected/341adb1c-5c4f-41ce-a0e4-d929d8b32121-kube-api-access-d9s42\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:43:50.191292 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:50.191257    4473 generic.go:356] "Generic (PLEG): container finished" podID="341adb1c-5c4f-41ce-a0e4-d929d8b32121" containerID="33463b1eb9bc5179227ce4525a53f2b38ab431eb09fe2d60c57dfce5eac627f1" exitCode=0
Nov 03 09:43:50.191487 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:50.191310    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872281-546f59c95-7tpjd"
Nov 03 09:43:50.191487 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:50.191349    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872281-546f59c95-7tpjd" event={"ID":"341adb1c-5c4f-41ce-a0e4-d929d8b32121","Type":"ContainerDied","Data":"33463b1eb9bc5179227ce4525a53f2b38ab431eb09fe2d60c57dfce5eac627f1"}
Nov 03 09:43:50.191487 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:50.191417    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872281-546f59c95-7tpjd" event={"ID":"341adb1c-5c4f-41ce-a0e4-d929d8b32121","Type":"ContainerDied","Data":"7e1026e6eaa055550bbc29adf8eb18c22c5cd1d4ca8bb830b71972a2804365b6"}
Nov 03 09:43:50.191487 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:50.191444    4473 scope.go:122] "RemoveContainer" containerID="33463b1eb9bc5179227ce4525a53f2b38ab431eb09fe2d60c57dfce5eac627f1"
Nov 03 09:43:50.192776 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:50.192746    4473 generic.go:356] "Generic (PLEG): container finished" podID="c4e5a6e1-68ac-4090-805f-e0a944f505b9" containerID="8d93dd4061a6eb49b43563f03a4d294ece33590d5356fdd122c70156da14da23" exitCode=0
Nov 03 09:43:50.192904 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:50.192800    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-872281-5d9f945fc-wnl4l"
Nov 03 09:43:50.192904 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:50.192797    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-872281-5d9f945fc-wnl4l" event={"ID":"c4e5a6e1-68ac-4090-805f-e0a944f505b9","Type":"ContainerDied","Data":"8d93dd4061a6eb49b43563f03a4d294ece33590d5356fdd122c70156da14da23"}
Nov 03 09:43:50.192904 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:50.192892    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-872281-5d9f945fc-wnl4l" event={"ID":"c4e5a6e1-68ac-4090-805f-e0a944f505b9","Type":"ContainerDied","Data":"312b04ce7e3632dc4b323ff9eb33d6da13fe965403e09057430ade16513f71a7"}
Nov 03 09:43:50.201594 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:50.201580    4473 scope.go:122] "RemoveContainer" containerID="33463b1eb9bc5179227ce4525a53f2b38ab431eb09fe2d60c57dfce5eac627f1"
Nov 03 09:43:50.201835 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:43:50.201815    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33463b1eb9bc5179227ce4525a53f2b38ab431eb09fe2d60c57dfce5eac627f1\": container with ID starting with 33463b1eb9bc5179227ce4525a53f2b38ab431eb09fe2d60c57dfce5eac627f1 not found: ID does not exist" containerID="33463b1eb9bc5179227ce4525a53f2b38ab431eb09fe2d60c57dfce5eac627f1"
Nov 03 09:43:50.201906 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:50.201843    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33463b1eb9bc5179227ce4525a53f2b38ab431eb09fe2d60c57dfce5eac627f1"} err="failed to get container status \"33463b1eb9bc5179227ce4525a53f2b38ab431eb09fe2d60c57dfce5eac627f1\": rpc error: code = NotFound desc = could not find container \"33463b1eb9bc5179227ce4525a53f2b38ab431eb09fe2d60c57dfce5eac627f1\": container with ID starting with 33463b1eb9bc5179227ce4525a53f2b38ab431eb09fe2d60c57dfce5eac627f1 not found: ID does not exist"
Nov 03 09:43:50.201906 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:50.201861    4473 scope.go:122] "RemoveContainer" containerID="8d93dd4061a6eb49b43563f03a4d294ece33590d5356fdd122c70156da14da23"
Nov 03 09:43:50.210544 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:50.210525    4473 scope.go:122] "RemoveContainer" containerID="8d93dd4061a6eb49b43563f03a4d294ece33590d5356fdd122c70156da14da23"
Nov 03 09:43:50.210805 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:43:50.210786    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d93dd4061a6eb49b43563f03a4d294ece33590d5356fdd122c70156da14da23\": container with ID starting with 8d93dd4061a6eb49b43563f03a4d294ece33590d5356fdd122c70156da14da23 not found: ID does not exist" containerID="8d93dd4061a6eb49b43563f03a4d294ece33590d5356fdd122c70156da14da23"
Nov 03 09:43:50.210851 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:50.210811    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d93dd4061a6eb49b43563f03a4d294ece33590d5356fdd122c70156da14da23"} err="failed to get container status \"8d93dd4061a6eb49b43563f03a4d294ece33590d5356fdd122c70156da14da23\": rpc error: code = NotFound desc = could not find container \"8d93dd4061a6eb49b43563f03a4d294ece33590d5356fdd122c70156da14da23\": container with ID starting with 8d93dd4061a6eb49b43563f03a4d294ece33590d5356fdd122c70156da14da23 not found: ID does not exist"
Nov 03 09:43:50.359708 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:50.359635    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-872281-5d9f945fc-wnl4l"]
Nov 03 09:43:50.409792 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:50.409767    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-872281-5d9f945fc-wnl4l"]
Nov 03 09:43:50.479987 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:50.479933    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-872281-546f59c95-7tpjd"]
Nov 03 09:43:50.488939 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:50.488907    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-872281-546f59c95-7tpjd"]
Nov 03 09:43:51.871542 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:51.871511    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="341adb1c-5c4f-41ce-a0e4-d929d8b32121" path="/var/lib/kubelet/pods/341adb1c-5c4f-41ce-a0e4-d929d8b32121/volumes"
Nov 03 09:43:51.871932 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:51.871909    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4e5a6e1-68ac-4090-805f-e0a944f505b9" path="/var/lib/kubelet/pods/c4e5a6e1-68ac-4090-805f-e0a944f505b9/volumes"
Nov 03 09:43:55.523470 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:55.523437    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcb36718-ql7dd"]
Nov 03 09:43:55.523987 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:55.523974    4473 cpu_manager.go:414] "containerMap: removing container" podUID="f6930088-1658-49d2-a5a4-32870da3c0cc" containerName="payload-processing"
Nov 03 09:43:55.524046 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:55.523988    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6930088-1658-49d2-a5a4-32870da3c0cc" containerName="payload-processing"
Nov 03 09:43:55.524046 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:55.524002    4473 cpu_manager.go:414] "containerMap: removing container" podUID="341adb1c-5c4f-41ce-a0e4-d929d8b32121" containerName="payload-pre-processing"
Nov 03 09:43:55.524046 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:55.524009    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="341adb1c-5c4f-41ce-a0e4-d929d8b32121" containerName="payload-pre-processing"
Nov 03 09:43:55.524046 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:55.524018    4473 cpu_manager.go:414] "containerMap: removing container" podUID="d6f5b134-d645-43dc-a388-10e05dc52168" containerName="payload-pre-processing"
Nov 03 09:43:55.524046 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:55.524023    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6f5b134-d645-43dc-a388-10e05dc52168" containerName="payload-pre-processing"
Nov 03 09:43:55.524233 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:55.524050    4473 cpu_manager.go:414] "containerMap: removing container" podUID="c4e5a6e1-68ac-4090-805f-e0a944f505b9" containerName="payload-processing"
Nov 03 09:43:55.524233 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:55.524055    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4e5a6e1-68ac-4090-805f-e0a944f505b9" containerName="payload-processing"
Nov 03 09:43:55.524233 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:55.524064    4473 cpu_manager.go:414] "containerMap: removing container" podUID="0c195d47-54ec-435a-9a6d-9f5d5caa5131" containerName="payload-processing"
Nov 03 09:43:55.524233 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:55.524069    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c195d47-54ec-435a-9a6d-9f5d5caa5131" containerName="payload-processing"
Nov 03 09:43:55.524233 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:55.524077    4473 cpu_manager.go:414] "containerMap: removing container" podUID="a02abcb7-266e-40cd-a1fb-fca2b5706f14" containerName="payload-pre-processing"
Nov 03 09:43:55.524233 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:55.524081    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a02abcb7-266e-40cd-a1fb-fca2b5706f14" containerName="payload-pre-processing"
Nov 03 09:43:55.524233 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:55.524092    4473 cpu_manager.go:414] "containerMap: removing container" podUID="2ff5c62e-0eaa-45d9-86db-8636677353fc" containerName="payload-processing"
Nov 03 09:43:55.524233 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:55.524097    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ff5c62e-0eaa-45d9-86db-8636677353fc" containerName="payload-processing"
Nov 03 09:43:55.524233 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:55.524196    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ff5c62e-0eaa-45d9-86db-8636677353fc" containerName="payload-processing"
Nov 03 09:43:55.524233 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:55.524208    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6930088-1658-49d2-a5a4-32870da3c0cc" containerName="payload-processing"
Nov 03 09:43:55.524233 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:55.524216    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="341adb1c-5c4f-41ce-a0e4-d929d8b32121" containerName="payload-pre-processing"
Nov 03 09:43:55.524233 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:55.524222    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6f5b134-d645-43dc-a388-10e05dc52168" containerName="payload-pre-processing"
Nov 03 09:43:55.524233 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:55.524233    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4e5a6e1-68ac-4090-805f-e0a944f505b9" containerName="payload-processing"
Nov 03 09:43:55.524233 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:55.524239    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c195d47-54ec-435a-9a6d-9f5d5caa5131" containerName="payload-processing"
Nov 03 09:43:55.524697 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:55.524245    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="a02abcb7-266e-40cd-a1fb-fca2b5706f14" containerName="payload-pre-processing"
Nov 03 09:43:55.528992 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:55.528975    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb36718-ql7dd"
Nov 03 09:43:55.571927 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:55.571893    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45f126fe-f88c-43af-8c53-a24b0f891c25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcb36718-ql7dd\" (UID: \"45f126fe-f88c-43af-8c53-a24b0f891c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb36718-ql7dd"
Nov 03 09:43:55.572086 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:55.571933    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srgjt\" (UniqueName: \"kubernetes.io/projected/45f126fe-f88c-43af-8c53-a24b0f891c25-kube-api-access-srgjt\") pod \"maas-api-revoke-sub-fcb36718-ql7dd\" (UID: \"45f126fe-f88c-43af-8c53-a24b0f891c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb36718-ql7dd"
Nov 03 09:43:55.572086 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:55.572067    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45f126fe-f88c-43af-8c53-a24b0f891c25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcb36718-ql7dd\" (UID: \"45f126fe-f88c-43af-8c53-a24b0f891c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb36718-ql7dd"
Nov 03 09:43:55.584751 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:55.584721    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcb36718-ql7dd"]
Nov 03 09:43:55.631285 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:55.631259    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:43:55.631432 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:55.631289    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:43:55.631432 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:55.631333    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:43:55.632754 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:55.632740    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:43:55.673866 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:55.673836    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45f126fe-f88c-43af-8c53-a24b0f891c25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcb36718-ql7dd\" (UID: \"45f126fe-f88c-43af-8c53-a24b0f891c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb36718-ql7dd"
Nov 03 09:43:55.674042 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:55.673868    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srgjt\" (UniqueName: \"kubernetes.io/projected/45f126fe-f88c-43af-8c53-a24b0f891c25-kube-api-access-srgjt\") pod \"maas-api-revoke-sub-fcb36718-ql7dd\" (UID: \"45f126fe-f88c-43af-8c53-a24b0f891c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb36718-ql7dd"
Nov 03 09:43:55.674042 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:55.673895    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45f126fe-f88c-43af-8c53-a24b0f891c25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcb36718-ql7dd\" (UID: \"45f126fe-f88c-43af-8c53-a24b0f891c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb36718-ql7dd"
Nov 03 09:43:55.674485 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:55.674466    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45f126fe-f88c-43af-8c53-a24b0f891c25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcb36718-ql7dd\" (UID: \"45f126fe-f88c-43af-8c53-a24b0f891c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb36718-ql7dd"
Nov 03 09:43:55.676409 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:55.676389    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45f126fe-f88c-43af-8c53-a24b0f891c25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcb36718-ql7dd\" (UID: \"45f126fe-f88c-43af-8c53-a24b0f891c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb36718-ql7dd"
Nov 03 09:43:55.732355 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:55.732335    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srgjt\" (UniqueName: \"kubernetes.io/projected/45f126fe-f88c-43af-8c53-a24b0f891c25-kube-api-access-srgjt\") pod \"maas-api-revoke-sub-fcb36718-ql7dd\" (UID: \"45f126fe-f88c-43af-8c53-a24b0f891c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb36718-ql7dd"
Nov 03 09:43:55.841975 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:55.841882    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb36718-ql7dd"
Nov 03 09:43:56.218917 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:56.218889    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcb36718-ql7dd"]
Nov 03 09:43:56.219773 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:43:56.219741    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45f126fe_f88c_43af_8c53_a24b0f891c25.slice/crio-ff8ef7833f1e8acf9a1353022a45f22493e5b4d5d6dfb964e5029ff74564ba4f WatchSource:0}: Error finding container ff8ef7833f1e8acf9a1353022a45f22493e5b4d5d6dfb964e5029ff74564ba4f: Status 404 returned error can't find the container with id ff8ef7833f1e8acf9a1353022a45f22493e5b4d5d6dfb964e5029ff74564ba4f
Nov 03 09:43:56.233558 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:56.233530    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb36718-ql7dd" event={"ID":"45f126fe-f88c-43af-8c53-a24b0f891c25","Type":"ContainerStarted","Data":"ff8ef7833f1e8acf9a1353022a45f22493e5b4d5d6dfb964e5029ff74564ba4f"}
Nov 03 09:43:57.242229 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:57.242198    4473 generic.go:356] "Generic (PLEG): container finished" podID="45f126fe-f88c-43af-8c53-a24b0f891c25" containerID="d058e599b9ee318faa84389936b48a805a2a9910c56b8a0f4fe27c697812a2bf" exitCode=0
Nov 03 09:43:57.242651 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:57.242283    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb36718-ql7dd" event={"ID":"45f126fe-f88c-43af-8c53-a24b0f891c25","Type":"ContainerDied","Data":"d058e599b9ee318faa84389936b48a805a2a9910c56b8a0f4fe27c697812a2bf"}
Nov 03 09:43:58.383127 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:58.383101    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb36718-ql7dd"
Nov 03 09:43:58.506708 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:58.506620    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45f126fe-f88c-43af-8c53-a24b0f891c25-kube-api-access-srgjt\" (UniqueName: \"kubernetes.io/projected/45f126fe-f88c-43af-8c53-a24b0f891c25-kube-api-access-srgjt\") pod \"45f126fe-f88c-43af-8c53-a24b0f891c25\" (UID: \"45f126fe-f88c-43af-8c53-a24b0f891c25\") "
Nov 03 09:43:58.506708 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:58.506691    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45f126fe-f88c-43af-8c53-a24b0f891c25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45f126fe-f88c-43af-8c53-a24b0f891c25-maas-api-openshift-ca\") pod \"45f126fe-f88c-43af-8c53-a24b0f891c25\" (UID: \"45f126fe-f88c-43af-8c53-a24b0f891c25\") "
Nov 03 09:43:58.506972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:58.506742    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45f126fe-f88c-43af-8c53-a24b0f891c25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45f126fe-f88c-43af-8c53-a24b0f891c25-maas-api-xks-ca\") pod \"45f126fe-f88c-43af-8c53-a24b0f891c25\" (UID: \"45f126fe-f88c-43af-8c53-a24b0f891c25\") "
Nov 03 09:43:58.507160 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:58.507135    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45f126fe-f88c-43af-8c53-a24b0f891c25-maas-api-openshift-ca" pod "45f126fe-f88c-43af-8c53-a24b0f891c25" (UID: "45f126fe-f88c-43af-8c53-a24b0f891c25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:43:58.509411 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:58.509384    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45f126fe-f88c-43af-8c53-a24b0f891c25-maas-api-xks-ca" pod "45f126fe-f88c-43af-8c53-a24b0f891c25" (UID: "45f126fe-f88c-43af-8c53-a24b0f891c25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:43:58.509680 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:58.509645    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45f126fe-f88c-43af-8c53-a24b0f891c25-kube-api-access-srgjt" pod "45f126fe-f88c-43af-8c53-a24b0f891c25" (UID: "45f126fe-f88c-43af-8c53-a24b0f891c25"). InnerVolumeSpecName "kube-api-access-srgjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:43:58.608780 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:58.608749    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45f126fe-f88c-43af-8c53-a24b0f891c25-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:43:58.608780 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:58.608776    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45f126fe-f88c-43af-8c53-a24b0f891c25-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:43:58.608780 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:58.608787    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srgjt\" (UniqueName: \"kubernetes.io/projected/45f126fe-f88c-43af-8c53-a24b0f891c25-kube-api-access-srgjt\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:43:59.255070 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:59.255037    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb36718-ql7dd"
Nov 03 09:43:59.255070 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:59.255062    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcb36718-ql7dd" event={"ID":"45f126fe-f88c-43af-8c53-a24b0f891c25","Type":"ContainerDied","Data":"ff8ef7833f1e8acf9a1353022a45f22493e5b4d5d6dfb964e5029ff74564ba4f"}
Nov 03 09:43:59.255320 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:59.255113    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff8ef7833f1e8acf9a1353022a45f22493e5b4d5d6dfb964e5029ff74564ba4f"
Nov 03 09:43:59.864308 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:43:59.864274    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g" podUID="4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:44:06.902840 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:06.902800    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2646e5c6-7kzw9"]
Nov 03 09:44:06.903394 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:06.903379    4473 cpu_manager.go:414] "containerMap: removing container" podUID="45f126fe-f88c-43af-8c53-a24b0f891c25" containerName="revoke-keys"
Nov 03 09:44:06.903439 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:06.903395    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45f126fe-f88c-43af-8c53-a24b0f891c25" containerName="revoke-keys"
Nov 03 09:44:06.903522 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:06.903512    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="45f126fe-f88c-43af-8c53-a24b0f891c25" containerName="revoke-keys"
Nov 03 09:44:06.907062 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:06.907042    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2646e5c6-7kzw9"
Nov 03 09:44:06.962156 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:06.962127    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:44:06.962156 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:06.962126    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:44:07.006837 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:07.005674    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9nsl\" (UniqueName: \"kubernetes.io/projected/6958024a-dbcc-4d7f-80b9-c8ec8314ea52-kube-api-access-s9nsl\") pod \"maas-api-revoke-sub-2646e5c6-7kzw9\" (UID: \"6958024a-dbcc-4d7f-80b9-c8ec8314ea52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2646e5c6-7kzw9"
Nov 03 09:44:07.006837 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:07.005807    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6958024a-dbcc-4d7f-80b9-c8ec8314ea52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2646e5c6-7kzw9\" (UID: \"6958024a-dbcc-4d7f-80b9-c8ec8314ea52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2646e5c6-7kzw9"
Nov 03 09:44:07.006837 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:07.005840    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6958024a-dbcc-4d7f-80b9-c8ec8314ea52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2646e5c6-7kzw9\" (UID: \"6958024a-dbcc-4d7f-80b9-c8ec8314ea52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2646e5c6-7kzw9"
Nov 03 09:44:07.026995 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:07.026011    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:44:07.026995 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:07.026135    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:44:07.035129 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:07.035051    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2646e5c6-7kzw9"]
Nov 03 09:44:07.107365 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:07.107333    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9nsl\" (UniqueName: \"kubernetes.io/projected/6958024a-dbcc-4d7f-80b9-c8ec8314ea52-kube-api-access-s9nsl\") pod \"maas-api-revoke-sub-2646e5c6-7kzw9\" (UID: \"6958024a-dbcc-4d7f-80b9-c8ec8314ea52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2646e5c6-7kzw9"
Nov 03 09:44:07.107578 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:07.107412    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6958024a-dbcc-4d7f-80b9-c8ec8314ea52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2646e5c6-7kzw9\" (UID: \"6958024a-dbcc-4d7f-80b9-c8ec8314ea52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2646e5c6-7kzw9"
Nov 03 09:44:07.107578 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:07.107441    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6958024a-dbcc-4d7f-80b9-c8ec8314ea52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2646e5c6-7kzw9\" (UID: \"6958024a-dbcc-4d7f-80b9-c8ec8314ea52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2646e5c6-7kzw9"
Nov 03 09:44:07.108160 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:07.108135    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6958024a-dbcc-4d7f-80b9-c8ec8314ea52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2646e5c6-7kzw9\" (UID: \"6958024a-dbcc-4d7f-80b9-c8ec8314ea52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2646e5c6-7kzw9"
Nov 03 09:44:07.110051 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:07.110027    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6958024a-dbcc-4d7f-80b9-c8ec8314ea52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2646e5c6-7kzw9\" (UID: \"6958024a-dbcc-4d7f-80b9-c8ec8314ea52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2646e5c6-7kzw9"
Nov 03 09:44:07.167245 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:07.167169    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9nsl\" (UniqueName: \"kubernetes.io/projected/6958024a-dbcc-4d7f-80b9-c8ec8314ea52-kube-api-access-s9nsl\") pod \"maas-api-revoke-sub-2646e5c6-7kzw9\" (UID: \"6958024a-dbcc-4d7f-80b9-c8ec8314ea52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2646e5c6-7kzw9"
Nov 03 09:44:07.222805 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:07.222758    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2646e5c6-7kzw9"
Nov 03 09:44:07.599231 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:07.599204    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2646e5c6-7kzw9"]
Nov 03 09:44:07.600293 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:44:07.600264    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6958024a_dbcc_4d7f_80b9_c8ec8314ea52.slice/crio-a29a49aa561ada76d12fa3a778346b51e679a513e11882d30e25caa562950144 WatchSource:0}: Error finding container a29a49aa561ada76d12fa3a778346b51e679a513e11882d30e25caa562950144: Status 404 returned error can't find the container with id a29a49aa561ada76d12fa3a778346b51e679a513e11882d30e25caa562950144
Nov 03 09:44:08.310190 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:08.310112    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2646e5c6-7kzw9" event={"ID":"6958024a-dbcc-4d7f-80b9-c8ec8314ea52","Type":"ContainerStarted","Data":"83e94e8b67d401656ea6f82e13a699591962148c564b4a1ebd07d00c685b485f"}
Nov 03 09:44:08.310190 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:08.310190    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2646e5c6-7kzw9" event={"ID":"6958024a-dbcc-4d7f-80b9-c8ec8314ea52","Type":"ContainerStarted","Data":"a29a49aa561ada76d12fa3a778346b51e679a513e11882d30e25caa562950144"}
Nov 03 09:44:08.426441 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:08.426389    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2646e5c6-7kzw9" podStartSLOduration=2.426375179 podStartE2EDuration="2.426375179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:44:08.425636105 +0000 UTC m=+3872.694699581" watchObservedRunningTime="2026-11-03 09:44:08.426375179 +0000 UTC m=+3872.695438612"
Nov 03 09:44:09.317293 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:09.317258    4473 generic.go:356] "Generic (PLEG): container finished" podID="6958024a-dbcc-4d7f-80b9-c8ec8314ea52" containerID="83e94e8b67d401656ea6f82e13a699591962148c564b4a1ebd07d00c685b485f" exitCode=0
Nov 03 09:44:09.317730 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:09.317333    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2646e5c6-7kzw9" event={"ID":"6958024a-dbcc-4d7f-80b9-c8ec8314ea52","Type":"ContainerDied","Data":"83e94e8b67d401656ea6f82e13a699591962148c564b4a1ebd07d00c685b485f"}
Nov 03 09:44:09.864987 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:09.864928    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g" podUID="4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:44:10.052866 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:10.052835    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42d4f8b1-6lr8w"]
Nov 03 09:44:10.055474 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:10.055446    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42d4f8b1-6lr8w"]
Nov 03 09:44:10.506876 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:10.506848    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2646e5c6-7kzw9"
Nov 03 09:44:10.646658 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:10.646623    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6958024a-dbcc-4d7f-80b9-c8ec8314ea52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6958024a-dbcc-4d7f-80b9-c8ec8314ea52-maas-api-xks-ca\") pod \"6958024a-dbcc-4d7f-80b9-c8ec8314ea52\" (UID: \"6958024a-dbcc-4d7f-80b9-c8ec8314ea52\") "
Nov 03 09:44:10.646832 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:10.646781    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6958024a-dbcc-4d7f-80b9-c8ec8314ea52-kube-api-access-s9nsl\" (UniqueName: \"kubernetes.io/projected/6958024a-dbcc-4d7f-80b9-c8ec8314ea52-kube-api-access-s9nsl\") pod \"6958024a-dbcc-4d7f-80b9-c8ec8314ea52\" (UID: \"6958024a-dbcc-4d7f-80b9-c8ec8314ea52\") "
Nov 03 09:44:10.646885 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:10.646830    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6958024a-dbcc-4d7f-80b9-c8ec8314ea52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6958024a-dbcc-4d7f-80b9-c8ec8314ea52-maas-api-openshift-ca\") pod \"6958024a-dbcc-4d7f-80b9-c8ec8314ea52\" (UID: \"6958024a-dbcc-4d7f-80b9-c8ec8314ea52\") "
Nov 03 09:44:10.647233 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:10.647210    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6958024a-dbcc-4d7f-80b9-c8ec8314ea52-maas-api-openshift-ca" pod "6958024a-dbcc-4d7f-80b9-c8ec8314ea52" (UID: "6958024a-dbcc-4d7f-80b9-c8ec8314ea52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:44:10.647411 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:10.647388    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6958024a-dbcc-4d7f-80b9-c8ec8314ea52-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:44:10.649174 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:10.649152    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6958024a-dbcc-4d7f-80b9-c8ec8314ea52-maas-api-xks-ca" pod "6958024a-dbcc-4d7f-80b9-c8ec8314ea52" (UID: "6958024a-dbcc-4d7f-80b9-c8ec8314ea52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:44:10.649501 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:10.649477    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6958024a-dbcc-4d7f-80b9-c8ec8314ea52-kube-api-access-s9nsl" pod "6958024a-dbcc-4d7f-80b9-c8ec8314ea52" (UID: "6958024a-dbcc-4d7f-80b9-c8ec8314ea52"). InnerVolumeSpecName "kube-api-access-s9nsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:44:10.748192 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:10.748156    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9nsl\" (UniqueName: \"kubernetes.io/projected/6958024a-dbcc-4d7f-80b9-c8ec8314ea52-kube-api-access-s9nsl\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:44:10.748192 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:10.748188    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6958024a-dbcc-4d7f-80b9-c8ec8314ea52-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:44:11.329986 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:11.329935    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2646e5c6-7kzw9" event={"ID":"6958024a-dbcc-4d7f-80b9-c8ec8314ea52","Type":"ContainerDied","Data":"a29a49aa561ada76d12fa3a778346b51e679a513e11882d30e25caa562950144"}
Nov 03 09:44:11.330163 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:11.330009    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a29a49aa561ada76d12fa3a778346b51e679a513e11882d30e25caa562950144"
Nov 03 09:44:11.330163 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:11.329935    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2646e5c6-7kzw9"
Nov 03 09:44:11.870704 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:11.870672    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="271b0cc5-8c5a-422d-b48f-85766ccc2731" path="/var/lib/kubelet/pods/271b0cc5-8c5a-422d-b48f-85766ccc2731/volumes"
Nov 03 09:44:16.937860 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:44:16.937819    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:44:19.865279 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:19.865246    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g" podUID="4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:44:29.873855 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:29.873815    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g" podUID="4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:44:31.674285 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:31.674246    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g"]
Nov 03 09:44:31.675551 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:31.675524    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g" podUID="4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:44:36.681977 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:36.675525    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g" podUID="4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" containerName="payload-processing" containerID="cri-o://f03d4e5c798d582205062c9b91c3e87f35492054ddf5557cc8aa99dce4719ef4" gracePeriod=25
Nov 03 09:44:36.759009 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:36.758934    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 09:44:36.761516 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:36.761497    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 09:44:36.885911 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:36.885871    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-849fb7c5bb-n24w4"]
Nov 03 09:44:36.887026 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:36.887000    4473 cpu_manager.go:414] "containerMap: removing container" podUID="6958024a-dbcc-4d7f-80b9-c8ec8314ea52" containerName="revoke-keys"
Nov 03 09:44:36.887186 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:36.887171    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6958024a-dbcc-4d7f-80b9-c8ec8314ea52" containerName="revoke-keys"
Nov 03 09:44:36.887450 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:36.887432    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="6958024a-dbcc-4d7f-80b9-c8ec8314ea52" containerName="revoke-keys"
Nov 03 09:44:36.891583 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:36.891565    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-n24w4"
Nov 03 09:44:36.946298 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:36.946274    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-849fb7c5bb-n24w4"]
Nov 03 09:44:36.961457 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:36.961429    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g"
Nov 03 09:44:36.993357 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:36.993283    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-6cd2420cdeaf\""
Nov 03 09:44:36.993490 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:36.993404    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-bxfbl\""
Nov 03 09:44:37.054567 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:37.054538    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8-provider-credentials\") pod \"4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8\" (UID: \"4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8\") "
Nov 03 09:44:37.054716 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:37.054657    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8-plugins-config-volume\") pod \"4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8\" (UID: \"4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8\") "
Nov 03 09:44:37.054785 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:37.054743    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8-kube-api-access-rrr9c\" (UniqueName: \"kubernetes.io/projected/4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8-kube-api-access-rrr9c\") pod \"4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8\" (UID: \"4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8\") "
Nov 03 09:44:37.054849 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:37.054788    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8-routing-overlay-volume\") pod \"4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8\" (UID: \"4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8\") "
Nov 03 09:44:37.055086 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:37.055065    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c4831543-6013-4af5-b852-5aa931d30d26-routing-overlay-volume\") pod \"payload-processing-external-model-849fb7c5bb-n24w4\" (UID: \"c4831543-6013-4af5-b852-5aa931d30d26\") " pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-n24w4"
Nov 03 09:44:37.055196 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:37.055123    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtjzd\" (UniqueName: \"kubernetes.io/projected/c4831543-6013-4af5-b852-5aa931d30d26-kube-api-access-rtjzd\") pod \"payload-processing-external-model-849fb7c5bb-n24w4\" (UID: \"c4831543-6013-4af5-b852-5aa931d30d26\") " pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-n24w4"
Nov 03 09:44:37.055196 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:37.055174    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8-plugins-config-volume" pod "4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" (UID: "4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:44:37.055306 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:37.055224    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8-routing-overlay-volume" pod "4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" (UID: "4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:44:37.055351 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:37.055305    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c4831543-6013-4af5-b852-5aa931d30d26-provider-credentials\") pod \"payload-processing-external-model-849fb7c5bb-n24w4\" (UID: \"c4831543-6013-4af5-b852-5aa931d30d26\") " pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-n24w4"
Nov 03 09:44:37.055571 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:37.055554    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4831543-6013-4af5-b852-5aa931d30d26-plugins-config-volume\") pod \"payload-processing-external-model-849fb7c5bb-n24w4\" (UID: \"c4831543-6013-4af5-b852-5aa931d30d26\") " pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-n24w4"
Nov 03 09:44:37.055654 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:37.055639    4473 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8-routing-overlay-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:44:37.055694 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:37.055663    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:44:37.057115 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:37.057088    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8-kube-api-access-rrr9c" pod "4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" (UID: "4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8"). InnerVolumeSpecName "kube-api-access-rrr9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:44:37.057282 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:37.057264    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8-provider-credentials" pod "4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" (UID: "4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:44:37.157231 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:37.157200    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c4831543-6013-4af5-b852-5aa931d30d26-provider-credentials\") pod \"payload-processing-external-model-849fb7c5bb-n24w4\" (UID: \"c4831543-6013-4af5-b852-5aa931d30d26\") " pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-n24w4"
Nov 03 09:44:37.157408 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:37.157278    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4831543-6013-4af5-b852-5aa931d30d26-plugins-config-volume\") pod \"payload-processing-external-model-849fb7c5bb-n24w4\" (UID: \"c4831543-6013-4af5-b852-5aa931d30d26\") " pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-n24w4"
Nov 03 09:44:37.157408 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:37.157353    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c4831543-6013-4af5-b852-5aa931d30d26-routing-overlay-volume\") pod \"payload-processing-external-model-849fb7c5bb-n24w4\" (UID: \"c4831543-6013-4af5-b852-5aa931d30d26\") " pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-n24w4"
Nov 03 09:44:37.157408 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:37.157390    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtjzd\" (UniqueName: \"kubernetes.io/projected/c4831543-6013-4af5-b852-5aa931d30d26-kube-api-access-rtjzd\") pod \"payload-processing-external-model-849fb7c5bb-n24w4\" (UID: \"c4831543-6013-4af5-b852-5aa931d30d26\") " pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-n24w4"
Nov 03 09:44:37.157570 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:37.157488    4473 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8-provider-credentials\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:44:37.157570 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:37.157506    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrr9c\" (UniqueName: \"kubernetes.io/projected/4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8-kube-api-access-rrr9c\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:44:37.158139 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:37.158105    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c4831543-6013-4af5-b852-5aa931d30d26-routing-overlay-volume\") pod \"payload-processing-external-model-849fb7c5bb-n24w4\" (UID: \"c4831543-6013-4af5-b852-5aa931d30d26\") " pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-n24w4"
Nov 03 09:44:37.158599 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:37.158571    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4831543-6013-4af5-b852-5aa931d30d26-plugins-config-volume\") pod \"payload-processing-external-model-849fb7c5bb-n24w4\" (UID: \"c4831543-6013-4af5-b852-5aa931d30d26\") " pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-n24w4"
Nov 03 09:44:37.160020 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:37.159998    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c4831543-6013-4af5-b852-5aa931d30d26-provider-credentials\") pod \"payload-processing-external-model-849fb7c5bb-n24w4\" (UID: \"c4831543-6013-4af5-b852-5aa931d30d26\") " pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-n24w4"
Nov 03 09:44:37.216927 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:37.216858    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtjzd\" (UniqueName: \"kubernetes.io/projected/c4831543-6013-4af5-b852-5aa931d30d26-kube-api-access-rtjzd\") pod \"payload-processing-external-model-849fb7c5bb-n24w4\" (UID: \"c4831543-6013-4af5-b852-5aa931d30d26\") " pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-n24w4"
Nov 03 09:44:37.511093 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:37.511010    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-n24w4"
Nov 03 09:44:37.616060 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:37.616024    4473 generic.go:356] "Generic (PLEG): container finished" podID="4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" containerID="f03d4e5c798d582205062c9b91c3e87f35492054ddf5557cc8aa99dce4719ef4" exitCode=0
Nov 03 09:44:37.616239 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:37.616081    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g"
Nov 03 09:44:37.616239 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:37.616110    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g" event={"ID":"4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8","Type":"ContainerDied","Data":"f03d4e5c798d582205062c9b91c3e87f35492054ddf5557cc8aa99dce4719ef4"}
Nov 03 09:44:37.616239 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:37.616200    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g" event={"ID":"4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8","Type":"ContainerDied","Data":"53738e0eaacc8c97a80965d922c37800953c39f936b290061de8e6f92b58db0b"}
Nov 03 09:44:37.616239 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:37.616229    4473 scope.go:122] "RemoveContainer" containerID="f03d4e5c798d582205062c9b91c3e87f35492054ddf5557cc8aa99dce4719ef4"
Nov 03 09:44:37.627954 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:37.627863    4473 scope.go:122] "RemoveContainer" containerID="f03d4e5c798d582205062c9b91c3e87f35492054ddf5557cc8aa99dce4719ef4"
Nov 03 09:44:37.628149 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:44:37.628127    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f03d4e5c798d582205062c9b91c3e87f35492054ddf5557cc8aa99dce4719ef4\": container with ID starting with f03d4e5c798d582205062c9b91c3e87f35492054ddf5557cc8aa99dce4719ef4 not found: ID does not exist" containerID="f03d4e5c798d582205062c9b91c3e87f35492054ddf5557cc8aa99dce4719ef4"
Nov 03 09:44:37.628229 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:37.628160    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f03d4e5c798d582205062c9b91c3e87f35492054ddf5557cc8aa99dce4719ef4"} err="failed to get container status \"f03d4e5c798d582205062c9b91c3e87f35492054ddf5557cc8aa99dce4719ef4\": rpc error: code = NotFound desc = could not find container \"f03d4e5c798d582205062c9b91c3e87f35492054ddf5557cc8aa99dce4719ef4\": container with ID starting with f03d4e5c798d582205062c9b91c3e87f35492054ddf5557cc8aa99dce4719ef4 not found: ID does not exist"
Nov 03 09:44:37.785504 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:37.785438    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g"]
Nov 03 09:44:37.835505 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:37.835475    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-66db7f95d6-dtd8g"]
Nov 03 09:44:37.871013 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:37.870983    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" path="/var/lib/kubelet/pods/4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8/volumes"
Nov 03 09:44:37.948915 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:37.948887    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-849fb7c5bb-n24w4"]
Nov 03 09:44:38.624922 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:38.624880    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-n24w4" event={"ID":"c4831543-6013-4af5-b852-5aa931d30d26","Type":"ContainerStarted","Data":"97b4a976b660cfe3ba3cb50dab1e1d66fc0d8df15ad128b018b6555ae5399608"}
Nov 03 09:44:38.625155 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:38.625105    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-n24w4"
Nov 03 09:44:38.625155 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:38.625145    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-n24w4" event={"ID":"c4831543-6013-4af5-b852-5aa931d30d26","Type":"ContainerStarted","Data":"a059062869b20727531196f3f7b92e3aa574ef73025d6dfeb3802faf5abed1e2"}
Nov 03 09:44:38.740762 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:38.740672    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-n24w4" podStartSLOduration=2.740655088 podStartE2EDuration="2.740655088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:44:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:44:38.740255741 +0000 UTC m=+3903.009319174" watchObservedRunningTime="2026-11-03 09:44:38.740655088 +0000 UTC m=+3903.009718523"
Nov 03 09:44:38.784837 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:44:38.784806    4473 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6cd2420cdeaf: secret "agc-cred-6cd2420cdeaf" not found
Nov 03 09:44:38.784837 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:44:38.784835    4473 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-849fb7c5bb-n24w4: secret "agc-cred-6cd2420cdeaf" not found
Nov 03 09:44:38.785088 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:44:38.784919    4473 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4831543-6013-4af5-b852-5aa931d30d26-provider-credentials podName:c4831543-6013-4af5-b852-5aa931d30d26 nodeName:}" failed. No retries permitted until 2026-11-03 09:44:39.284897156 +0000 UTC m=+3903.553960572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c4831543-6013-4af5-b852-5aa931d30d26-provider-credentials") pod "payload-processing-external-model-849fb7c5bb-n24w4" (UID: "c4831543-6013-4af5-b852-5aa931d30d26") : secret "agc-cred-6cd2420cdeaf" not found
Nov 03 09:44:39.313578 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:44:39.304257    4473 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6cd2420cdeaf: secret "agc-cred-6cd2420cdeaf" not found
Nov 03 09:44:39.313578 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:44:39.304289    4473 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-849fb7c5bb-n24w4: secret "agc-cred-6cd2420cdeaf" not found
Nov 03 09:44:39.313578 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:44:39.304353    4473 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4831543-6013-4af5-b852-5aa931d30d26-provider-credentials podName:c4831543-6013-4af5-b852-5aa931d30d26 nodeName:}" failed. No retries permitted until 2026-11-03 09:44:40.304333608 +0000 UTC m=+3904.573397043 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c4831543-6013-4af5-b852-5aa931d30d26-provider-credentials") pod "payload-processing-external-model-849fb7c5bb-n24w4" (UID: "c4831543-6013-4af5-b852-5aa931d30d26") : secret "agc-cred-6cd2420cdeaf" not found
Nov 03 09:44:40.314656 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:44:40.314628    4473 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6cd2420cdeaf: secret "agc-cred-6cd2420cdeaf" not found
Nov 03 09:44:40.314656 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:44:40.314650    4473 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-849fb7c5bb-n24w4: secret "agc-cred-6cd2420cdeaf" not found
Nov 03 09:44:40.315131 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:44:40.314703    4473 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4831543-6013-4af5-b852-5aa931d30d26-provider-credentials podName:c4831543-6013-4af5-b852-5aa931d30d26 nodeName:}" failed. No retries permitted until 2026-11-03 09:44:42.314689028 +0000 UTC m=+3906.583752453 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c4831543-6013-4af5-b852-5aa931d30d26-provider-credentials") pod "payload-processing-external-model-849fb7c5bb-n24w4" (UID: "c4831543-6013-4af5-b852-5aa931d30d26") : secret "agc-cred-6cd2420cdeaf" not found
Nov 03 09:44:42.337754 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:44:42.337724    4473 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6cd2420cdeaf: secret "agc-cred-6cd2420cdeaf" not found
Nov 03 09:44:42.337754 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:44:42.337748    4473 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-849fb7c5bb-n24w4: secret "agc-cred-6cd2420cdeaf" not found
Nov 03 09:44:42.338282 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:44:42.337799    4473 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4831543-6013-4af5-b852-5aa931d30d26-provider-credentials podName:c4831543-6013-4af5-b852-5aa931d30d26 nodeName:}" failed. No retries permitted until 2026-11-03 09:44:46.337784812 +0000 UTC m=+3910.606848238 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c4831543-6013-4af5-b852-5aa931d30d26-provider-credentials") pod "payload-processing-external-model-849fb7c5bb-n24w4" (UID: "c4831543-6013-4af5-b852-5aa931d30d26") : secret "agc-cred-6cd2420cdeaf" not found
Nov 03 09:44:43.342906 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:43.342864    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-849fb7c5bb-n24w4"]
Nov 03 09:44:43.346077 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:43.346053    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-n24w4"
Nov 03 09:44:44.380440 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:44.380393    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-dc1444-67d5485fc7-l8lqz"]
Nov 03 09:44:44.384560 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:44.384082    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-dc1444-6846f74759-89k9t"]
Nov 03 09:44:44.821537 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:44.821445    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:44:46.946432 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:44:46.946335    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:44:48.348058 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:48.348009    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-n24w4" podUID="c4831543-6013-4af5-b852-5aa931d30d26" containerName="payload-processing" containerID="cri-o://97b4a976b660cfe3ba3cb50dab1e1d66fc0d8df15ad128b018b6555ae5399608" gracePeriod=25
Nov 03 09:44:48.712388 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:48.712357    4473 generic.go:356] "Generic (PLEG): container finished" podID="c4831543-6013-4af5-b852-5aa931d30d26" containerID="97b4a976b660cfe3ba3cb50dab1e1d66fc0d8df15ad128b018b6555ae5399608" exitCode=0
Nov 03 09:44:48.712598 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:48.712431    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-n24w4" event={"ID":"c4831543-6013-4af5-b852-5aa931d30d26","Type":"ContainerDied","Data":"97b4a976b660cfe3ba3cb50dab1e1d66fc0d8df15ad128b018b6555ae5399608"}
Nov 03 09:44:48.909183 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:48.909119    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-n24w4"
Nov 03 09:44:48.997431 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:48.997396    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4831543-6013-4af5-b852-5aa931d30d26-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c4831543-6013-4af5-b852-5aa931d30d26-provider-credentials\") pod \"c4831543-6013-4af5-b852-5aa931d30d26\" (UID: \"c4831543-6013-4af5-b852-5aa931d30d26\") "
Nov 03 09:44:48.997613 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:48.997502    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4831543-6013-4af5-b852-5aa931d30d26-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c4831543-6013-4af5-b852-5aa931d30d26-routing-overlay-volume\") pod \"c4831543-6013-4af5-b852-5aa931d30d26\" (UID: \"c4831543-6013-4af5-b852-5aa931d30d26\") "
Nov 03 09:44:48.997658 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:48.997618    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4831543-6013-4af5-b852-5aa931d30d26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4831543-6013-4af5-b852-5aa931d30d26-plugins-config-volume\") pod \"c4831543-6013-4af5-b852-5aa931d30d26\" (UID: \"c4831543-6013-4af5-b852-5aa931d30d26\") "
Nov 03 09:44:48.997921 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:48.997885    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4831543-6013-4af5-b852-5aa931d30d26-routing-overlay-volume" pod "c4831543-6013-4af5-b852-5aa931d30d26" (UID: "c4831543-6013-4af5-b852-5aa931d30d26"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:44:48.998085 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:48.998062    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4831543-6013-4af5-b852-5aa931d30d26-plugins-config-volume" pod "c4831543-6013-4af5-b852-5aa931d30d26" (UID: "c4831543-6013-4af5-b852-5aa931d30d26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:44:48.998216 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:48.998195    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4831543-6013-4af5-b852-5aa931d30d26-kube-api-access-rtjzd\" (UniqueName: \"kubernetes.io/projected/c4831543-6013-4af5-b852-5aa931d30d26-kube-api-access-rtjzd\") pod \"c4831543-6013-4af5-b852-5aa931d30d26\" (UID: \"c4831543-6013-4af5-b852-5aa931d30d26\") "
Nov 03 09:44:48.998763 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:48.998739    4473 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c4831543-6013-4af5-b852-5aa931d30d26-routing-overlay-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:44:48.998763 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:48.998766    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4831543-6013-4af5-b852-5aa931d30d26-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:44:48.999851 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:48.999826    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4831543-6013-4af5-b852-5aa931d30d26-provider-credentials" pod "c4831543-6013-4af5-b852-5aa931d30d26" (UID: "c4831543-6013-4af5-b852-5aa931d30d26"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:44:49.000405 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:49.000380    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4831543-6013-4af5-b852-5aa931d30d26-kube-api-access-rtjzd" pod "c4831543-6013-4af5-b852-5aa931d30d26" (UID: "c4831543-6013-4af5-b852-5aa931d30d26"). InnerVolumeSpecName "kube-api-access-rtjzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:44:49.099636 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:49.099596    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtjzd\" (UniqueName: \"kubernetes.io/projected/c4831543-6013-4af5-b852-5aa931d30d26-kube-api-access-rtjzd\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:44:49.099636 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:49.099628    4473 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c4831543-6013-4af5-b852-5aa931d30d26-provider-credentials\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:44:49.380811 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:49.380763    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-dc1444-67d5485fc7-l8lqz" podUID="2fca6353-9585-4aa2-b9b0-395f9e0f0cce" containerName="payload-processing" containerID="cri-o://3a3303d5606fb25bcb7d69d8c0d8e370d48ee0fd27dc91d8539c05737a1c9f5a" gracePeriod=25
Nov 03 09:44:49.385172 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:49.385142    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc1444-6846f74759-89k9t" podUID="3bdce5e0-9b59-48dc-a4a6-851d72272b46" containerName="payload-pre-processing" containerID="cri-o://98a70ddc3ddd342ff067dceb43456d566c45e6d7519c70244fe16ddba6f72da4" gracePeriod=25
Nov 03 09:44:49.658501 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:49.658481    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-dc1444-67d5485fc7-l8lqz"
Nov 03 09:44:49.661484 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:49.661464    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc1444-6846f74759-89k9t"
Nov 03 09:44:49.719187 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:49.719153    4473 generic.go:356] "Generic (PLEG): container finished" podID="2fca6353-9585-4aa2-b9b0-395f9e0f0cce" containerID="3a3303d5606fb25bcb7d69d8c0d8e370d48ee0fd27dc91d8539c05737a1c9f5a" exitCode=0
Nov 03 09:44:49.719342 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:49.719204    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-dc1444-67d5485fc7-l8lqz"
Nov 03 09:44:49.719342 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:49.719243    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-dc1444-67d5485fc7-l8lqz" event={"ID":"2fca6353-9585-4aa2-b9b0-395f9e0f0cce","Type":"ContainerDied","Data":"3a3303d5606fb25bcb7d69d8c0d8e370d48ee0fd27dc91d8539c05737a1c9f5a"}
Nov 03 09:44:49.719342 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:49.719309    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-dc1444-67d5485fc7-l8lqz" event={"ID":"2fca6353-9585-4aa2-b9b0-395f9e0f0cce","Type":"ContainerDied","Data":"f38f82ffa762afb438b225036e5fa834e767ed5af82adf41ef32fd9361947302"}
Nov 03 09:44:49.719482 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:49.719344    4473 scope.go:122] "RemoveContainer" containerID="3a3303d5606fb25bcb7d69d8c0d8e370d48ee0fd27dc91d8539c05737a1c9f5a"
Nov 03 09:44:49.720546 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:49.720526    4473 generic.go:356] "Generic (PLEG): container finished" podID="3bdce5e0-9b59-48dc-a4a6-851d72272b46" containerID="98a70ddc3ddd342ff067dceb43456d566c45e6d7519c70244fe16ddba6f72da4" exitCode=0
Nov 03 09:44:49.720640 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:49.720580    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc1444-6846f74759-89k9t"
Nov 03 09:44:49.720640 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:49.720616    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc1444-6846f74759-89k9t" event={"ID":"3bdce5e0-9b59-48dc-a4a6-851d72272b46","Type":"ContainerDied","Data":"98a70ddc3ddd342ff067dceb43456d566c45e6d7519c70244fe16ddba6f72da4"}
Nov 03 09:44:49.720929 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:49.720659    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-dc1444-6846f74759-89k9t" event={"ID":"3bdce5e0-9b59-48dc-a4a6-851d72272b46","Type":"ContainerDied","Data":"b029a1a44b6264f0503d8ae1fab78615c9f79a89570a1ef67082ad928286a1bf"}
Nov 03 09:44:49.722118 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:49.722100    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-n24w4"
Nov 03 09:44:49.722225 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:49.722100    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-n24w4" event={"ID":"c4831543-6013-4af5-b852-5aa931d30d26","Type":"ContainerDied","Data":"a059062869b20727531196f3f7b92e3aa574ef73025d6dfeb3802faf5abed1e2"}
Nov 03 09:44:49.730140 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:49.730002    4473 scope.go:122] "RemoveContainer" containerID="3a3303d5606fb25bcb7d69d8c0d8e370d48ee0fd27dc91d8539c05737a1c9f5a"
Nov 03 09:44:49.730318 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:44:49.730297    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a3303d5606fb25bcb7d69d8c0d8e370d48ee0fd27dc91d8539c05737a1c9f5a\": container with ID starting with 3a3303d5606fb25bcb7d69d8c0d8e370d48ee0fd27dc91d8539c05737a1c9f5a not found: ID does not exist" containerID="3a3303d5606fb25bcb7d69d8c0d8e370d48ee0fd27dc91d8539c05737a1c9f5a"
Nov 03 09:44:49.730400 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:49.730328    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a3303d5606fb25bcb7d69d8c0d8e370d48ee0fd27dc91d8539c05737a1c9f5a"} err="failed to get container status \"3a3303d5606fb25bcb7d69d8c0d8e370d48ee0fd27dc91d8539c05737a1c9f5a\": rpc error: code = NotFound desc = could not find container \"3a3303d5606fb25bcb7d69d8c0d8e370d48ee0fd27dc91d8539c05737a1c9f5a\": container with ID starting with 3a3303d5606fb25bcb7d69d8c0d8e370d48ee0fd27dc91d8539c05737a1c9f5a not found: ID does not exist"
Nov 03 09:44:49.730400 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:49.730351    4473 scope.go:122] "RemoveContainer" containerID="98a70ddc3ddd342ff067dceb43456d566c45e6d7519c70244fe16ddba6f72da4"
Nov 03 09:44:49.742745 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:49.742669    4473 scope.go:122] "RemoveContainer" containerID="98a70ddc3ddd342ff067dceb43456d566c45e6d7519c70244fe16ddba6f72da4"
Nov 03 09:44:49.743048 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:44:49.743013    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98a70ddc3ddd342ff067dceb43456d566c45e6d7519c70244fe16ddba6f72da4\": container with ID starting with 98a70ddc3ddd342ff067dceb43456d566c45e6d7519c70244fe16ddba6f72da4 not found: ID does not exist" containerID="98a70ddc3ddd342ff067dceb43456d566c45e6d7519c70244fe16ddba6f72da4"
Nov 03 09:44:49.743116 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:49.743058    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98a70ddc3ddd342ff067dceb43456d566c45e6d7519c70244fe16ddba6f72da4"} err="failed to get container status \"98a70ddc3ddd342ff067dceb43456d566c45e6d7519c70244fe16ddba6f72da4\": rpc error: code = NotFound desc = could not find container \"98a70ddc3ddd342ff067dceb43456d566c45e6d7519c70244fe16ddba6f72da4\": container with ID starting with 98a70ddc3ddd342ff067dceb43456d566c45e6d7519c70244fe16ddba6f72da4 not found: ID does not exist"
Nov 03 09:44:49.743116 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:49.743078    4473 scope.go:122] "RemoveContainer" containerID="97b4a976b660cfe3ba3cb50dab1e1d66fc0d8df15ad128b018b6555ae5399608"
Nov 03 09:44:49.808423 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:49.808396    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fca6353-9585-4aa2-b9b0-395f9e0f0cce-kube-api-access-9wp2n\" (UniqueName: \"kubernetes.io/projected/2fca6353-9585-4aa2-b9b0-395f9e0f0cce-kube-api-access-9wp2n\") pod \"2fca6353-9585-4aa2-b9b0-395f9e0f0cce\" (UID: \"2fca6353-9585-4aa2-b9b0-395f9e0f0cce\") "
Nov 03 09:44:49.808585 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:49.808437    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fca6353-9585-4aa2-b9b0-395f9e0f0cce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fca6353-9585-4aa2-b9b0-395f9e0f0cce-plugins-config-volume\") pod \"2fca6353-9585-4aa2-b9b0-395f9e0f0cce\" (UID: \"2fca6353-9585-4aa2-b9b0-395f9e0f0cce\") "
Nov 03 09:44:49.808585 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:49.808491    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bdce5e0-9b59-48dc-a4a6-851d72272b46-kube-api-access-dn7w9\" (UniqueName: \"kubernetes.io/projected/3bdce5e0-9b59-48dc-a4a6-851d72272b46-kube-api-access-dn7w9\") pod \"3bdce5e0-9b59-48dc-a4a6-851d72272b46\" (UID: \"3bdce5e0-9b59-48dc-a4a6-851d72272b46\") "
Nov 03 09:44:49.808585 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:49.808572    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bdce5e0-9b59-48dc-a4a6-851d72272b46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bdce5e0-9b59-48dc-a4a6-851d72272b46-plugins-config-volume\") pod \"3bdce5e0-9b59-48dc-a4a6-851d72272b46\" (UID: \"3bdce5e0-9b59-48dc-a4a6-851d72272b46\") "
Nov 03 09:44:49.809073 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:49.809031    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fca6353-9585-4aa2-b9b0-395f9e0f0cce-plugins-config-volume" pod "2fca6353-9585-4aa2-b9b0-395f9e0f0cce" (UID: "2fca6353-9585-4aa2-b9b0-395f9e0f0cce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:44:49.809198 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:49.809122    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bdce5e0-9b59-48dc-a4a6-851d72272b46-plugins-config-volume" pod "3bdce5e0-9b59-48dc-a4a6-851d72272b46" (UID: "3bdce5e0-9b59-48dc-a4a6-851d72272b46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:44:49.810772 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:49.810750    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bdce5e0-9b59-48dc-a4a6-851d72272b46-kube-api-access-dn7w9" pod "3bdce5e0-9b59-48dc-a4a6-851d72272b46" (UID: "3bdce5e0-9b59-48dc-a4a6-851d72272b46"). InnerVolumeSpecName "kube-api-access-dn7w9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:44:49.811246 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:49.811223    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fca6353-9585-4aa2-b9b0-395f9e0f0cce-kube-api-access-9wp2n" pod "2fca6353-9585-4aa2-b9b0-395f9e0f0cce" (UID: "2fca6353-9585-4aa2-b9b0-395f9e0f0cce"). InnerVolumeSpecName "kube-api-access-9wp2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:44:49.910366 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:49.910276    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wp2n\" (UniqueName: \"kubernetes.io/projected/2fca6353-9585-4aa2-b9b0-395f9e0f0cce-kube-api-access-9wp2n\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:44:49.910366 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:49.910313    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fca6353-9585-4aa2-b9b0-395f9e0f0cce-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:44:49.910366 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:49.910330    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn7w9\" (UniqueName: \"kubernetes.io/projected/3bdce5e0-9b59-48dc-a4a6-851d72272b46-kube-api-access-dn7w9\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:44:49.910366 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:49.910346    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bdce5e0-9b59-48dc-a4a6-851d72272b46-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:44:50.007574 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:50.007540    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-849fb7c5bb-n24w4"]
Nov 03 09:44:50.013860 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:50.013833    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-849fb7c5bb-n24w4"]
Nov 03 09:44:50.178895 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:50.178815    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-dc1444-6846f74759-89k9t"]
Nov 03 09:44:50.187290 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:50.187259    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-dc1444-6846f74759-89k9t"]
Nov 03 09:44:50.299784 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:50.299754    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-dc1444-67d5485fc7-l8lqz"]
Nov 03 09:44:50.304988 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:50.304960    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-dc1444-67d5485fc7-l8lqz"]
Nov 03 09:44:51.871819 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:51.871783    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fca6353-9585-4aa2-b9b0-395f9e0f0cce" path="/var/lib/kubelet/pods/2fca6353-9585-4aa2-b9b0-395f9e0f0cce/volumes"
Nov 03 09:44:51.872321 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:51.872302    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bdce5e0-9b59-48dc-a4a6-851d72272b46" path="/var/lib/kubelet/pods/3bdce5e0-9b59-48dc-a4a6-851d72272b46/volumes"
Nov 03 09:44:51.872672 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:51.872658    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4831543-6013-4af5-b852-5aa931d30d26" path="/var/lib/kubelet/pods/c4831543-6013-4af5-b852-5aa931d30d26/volumes"
Nov 03 09:44:53.084395 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:53.084362    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-849fb7c5bb-bzq6t"]
Nov 03 09:44:53.084875 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:53.084860    4473 cpu_manager.go:414] "containerMap: removing container" podUID="4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" containerName="payload-processing"
Nov 03 09:44:53.084875 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:53.084876    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" containerName="payload-processing"
Nov 03 09:44:53.085001 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:53.084890    4473 cpu_manager.go:414] "containerMap: removing container" podUID="3bdce5e0-9b59-48dc-a4a6-851d72272b46" containerName="payload-pre-processing"
Nov 03 09:44:53.085001 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:53.084895    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bdce5e0-9b59-48dc-a4a6-851d72272b46" containerName="payload-pre-processing"
Nov 03 09:44:53.085001 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:53.084914    4473 cpu_manager.go:414] "containerMap: removing container" podUID="2fca6353-9585-4aa2-b9b0-395f9e0f0cce" containerName="payload-processing"
Nov 03 09:44:53.085001 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:53.084918    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fca6353-9585-4aa2-b9b0-395f9e0f0cce" containerName="payload-processing"
Nov 03 09:44:53.085001 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:53.084935    4473 cpu_manager.go:414] "containerMap: removing container" podUID="c4831543-6013-4af5-b852-5aa931d30d26" containerName="payload-processing"
Nov 03 09:44:53.085001 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:53.084940    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4831543-6013-4af5-b852-5aa931d30d26" containerName="payload-processing"
Nov 03 09:44:53.085197 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:53.085052    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ba5b3b1-ee8a-42f9-b45b-cebbfa148eb8" containerName="payload-processing"
Nov 03 09:44:53.085197 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:53.085061    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bdce5e0-9b59-48dc-a4a6-851d72272b46" containerName="payload-pre-processing"
Nov 03 09:44:53.085197 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:53.085069    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fca6353-9585-4aa2-b9b0-395f9e0f0cce" containerName="payload-processing"
Nov 03 09:44:53.085197 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:53.085079    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4831543-6013-4af5-b852-5aa931d30d26" containerName="payload-processing"
Nov 03 09:44:53.089758 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:53.089740    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-bzq6t"
Nov 03 09:44:53.147856 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:53.147825    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-849fb7c5bb-bzq6t"]
Nov 03 09:44:53.151356 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:53.151327    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5dbe18e-50c8-402f-9f14-96e484f8fe67-plugins-config-volume\") pod \"payload-processing-external-model-849fb7c5bb-bzq6t\" (UID: \"f5dbe18e-50c8-402f-9f14-96e484f8fe67\") " pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-bzq6t"
Nov 03 09:44:53.151497 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:53.151370    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q5np\" (UniqueName: \"kubernetes.io/projected/f5dbe18e-50c8-402f-9f14-96e484f8fe67-kube-api-access-2q5np\") pod \"payload-processing-external-model-849fb7c5bb-bzq6t\" (UID: \"f5dbe18e-50c8-402f-9f14-96e484f8fe67\") " pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-bzq6t"
Nov 03 09:44:53.151557 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:53.151516    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5dbe18e-50c8-402f-9f14-96e484f8fe67-routing-overlay-volume\") pod \"payload-processing-external-model-849fb7c5bb-bzq6t\" (UID: \"f5dbe18e-50c8-402f-9f14-96e484f8fe67\") " pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-bzq6t"
Nov 03 09:44:53.151611 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:53.151599    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5dbe18e-50c8-402f-9f14-96e484f8fe67-provider-credentials\") pod \"payload-processing-external-model-849fb7c5bb-bzq6t\" (UID: \"f5dbe18e-50c8-402f-9f14-96e484f8fe67\") " pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-bzq6t"
Nov 03 09:44:53.194012 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:53.193980    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-6cd2420cdeaf\""
Nov 03 09:44:53.194012 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:53.194006    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-5mc64\""
Nov 03 09:44:53.194012 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:53.193997    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 09:44:53.194263 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:53.193982    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 09:44:53.194263 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:53.194059    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 09:44:53.194263 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:53.194120    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 09:44:53.252789 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:53.252758    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5dbe18e-50c8-402f-9f14-96e484f8fe67-routing-overlay-volume\") pod \"payload-processing-external-model-849fb7c5bb-bzq6t\" (UID: \"f5dbe18e-50c8-402f-9f14-96e484f8fe67\") " pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-bzq6t"
Nov 03 09:44:53.252933 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:53.252850    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5dbe18e-50c8-402f-9f14-96e484f8fe67-provider-credentials\") pod \"payload-processing-external-model-849fb7c5bb-bzq6t\" (UID: \"f5dbe18e-50c8-402f-9f14-96e484f8fe67\") " pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-bzq6t"
Nov 03 09:44:53.253013 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:53.252990    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5dbe18e-50c8-402f-9f14-96e484f8fe67-plugins-config-volume\") pod \"payload-processing-external-model-849fb7c5bb-bzq6t\" (UID: \"f5dbe18e-50c8-402f-9f14-96e484f8fe67\") " pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-bzq6t"
Nov 03 09:44:53.253050 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:53.253015    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q5np\" (UniqueName: \"kubernetes.io/projected/f5dbe18e-50c8-402f-9f14-96e484f8fe67-kube-api-access-2q5np\") pod \"payload-processing-external-model-849fb7c5bb-bzq6t\" (UID: \"f5dbe18e-50c8-402f-9f14-96e484f8fe67\") " pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-bzq6t"
Nov 03 09:44:53.253603 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:53.253535    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5dbe18e-50c8-402f-9f14-96e484f8fe67-routing-overlay-volume\") pod \"payload-processing-external-model-849fb7c5bb-bzq6t\" (UID: \"f5dbe18e-50c8-402f-9f14-96e484f8fe67\") " pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-bzq6t"
Nov 03 09:44:53.253801 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:53.253717    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5dbe18e-50c8-402f-9f14-96e484f8fe67-plugins-config-volume\") pod \"payload-processing-external-model-849fb7c5bb-bzq6t\" (UID: \"f5dbe18e-50c8-402f-9f14-96e484f8fe67\") " pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-bzq6t"
Nov 03 09:44:53.255719 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:53.255697    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5dbe18e-50c8-402f-9f14-96e484f8fe67-provider-credentials\") pod \"payload-processing-external-model-849fb7c5bb-bzq6t\" (UID: \"f5dbe18e-50c8-402f-9f14-96e484f8fe67\") " pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-bzq6t"
Nov 03 09:44:53.311687 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:53.311663    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q5np\" (UniqueName: \"kubernetes.io/projected/f5dbe18e-50c8-402f-9f14-96e484f8fe67-kube-api-access-2q5np\") pod \"payload-processing-external-model-849fb7c5bb-bzq6t\" (UID: \"f5dbe18e-50c8-402f-9f14-96e484f8fe67\") " pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-bzq6t"
Nov 03 09:44:53.404564 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:53.404538    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-bzq6t"
Nov 03 09:44:53.781831 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:53.781797    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-849fb7c5bb-bzq6t"]
Nov 03 09:44:53.783151 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:44:53.783127    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5dbe18e_50c8_402f_9f14_96e484f8fe67.slice/crio-5cbe4db1a1783eeb0dbcd0605359d02dc879f8798b48efb042de92c594011a4e WatchSource:0}: Error finding container 5cbe4db1a1783eeb0dbcd0605359d02dc879f8798b48efb042de92c594011a4e: Status 404 returned error can't find the container with id 5cbe4db1a1783eeb0dbcd0605359d02dc879f8798b48efb042de92c594011a4e
Nov 03 09:44:54.765666 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:54.765629    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-bzq6t" event={"ID":"f5dbe18e-50c8-402f-9f14-96e484f8fe67","Type":"ContainerStarted","Data":"762e391f563bf8497230893492d866a2389b0f71a3e72050968f56973c7ed8ad"}
Nov 03 09:44:54.766126 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:54.765692    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-bzq6t" event={"ID":"f5dbe18e-50c8-402f-9f14-96e484f8fe67","Type":"ContainerStarted","Data":"5cbe4db1a1783eeb0dbcd0605359d02dc879f8798b48efb042de92c594011a4e"}
Nov 03 09:44:54.766126 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:54.765737    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-bzq6t"
Nov 03 09:44:54.884568 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:44:54.884517    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-bzq6t" podStartSLOduration=1.884504003 podStartE2EDuration="1.884504003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:44:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:44:54.88292476 +0000 UTC m=+3919.151988214" watchObservedRunningTime="2026-11-03 09:44:54.884504003 +0000 UTC m=+3919.153567436"
Nov 03 09:45:00.164430 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.164380    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894985-brvhf"]
Nov 03 09:45:00.169184 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.169157    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894985-jp75m"]
Nov 03 09:45:00.169350 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.169268    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894985-brvhf"
Nov 03 09:45:00.174122 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.174103    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af514d-29894985-gzb4m"]
Nov 03 09:45:00.174224 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.174203    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894985-jp75m"
Nov 03 09:45:00.178486 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.178466    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af514d-29894985-gzb4m"
Nov 03 09:45:00.214762 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.214727    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894985-cftcl"]
Nov 03 09:45:00.218901 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.218879    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872281-29894985-xvvlh"]
Nov 03 09:45:00.219052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.219032    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894985-cftcl"
Nov 03 09:45:00.222541 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.222521    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894985-9pscs"]
Nov 03 09:45:00.222685 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.222647    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872281-29894985-xvvlh"
Nov 03 09:45:00.226102 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.226080    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894985-7kqh9"]
Nov 03 09:45:00.226229 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.226190    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894985-9pscs"
Nov 03 09:45:00.230490 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.230473    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894985-wv4z2"]
Nov 03 09:45:00.230602 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.230581    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894985-7kqh9"
Nov 03 09:45:00.233985 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.233966    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894985-fblt9"]
Nov 03 09:45:00.234068 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.234043    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894985-wv4z2"
Nov 03 09:45:00.238387 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.238372    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894985-2j8r6"]
Nov 03 09:45:00.238480 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.238466    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894985-fblt9"
Nov 03 09:45:00.241890 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.241872    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894985-2j8r6"
Nov 03 09:45:00.262249 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.262226    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:45:00.262249 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.262237    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:45:00.262396 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.262287    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:45:00.265328 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.265308    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894985-flfp2"]
Nov 03 09:45:00.270182 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.270157    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894985-brvhf"]
Nov 03 09:45:00.270297 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.270268    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894985-flfp2"
Nov 03 09:45:00.271645 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.271613    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894985-jp75m"]
Nov 03 09:45:00.274377 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.274358    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894985-cftcl"]
Nov 03 09:45:00.277550 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.277529    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872281-29894985-xvvlh"]
Nov 03 09:45:00.319430 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.318716    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894985-9pscs"]
Nov 03 09:45:00.322781 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.322753    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894985-7kqh9"]
Nov 03 09:45:00.344193 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.344160    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be2cbd18-00fc-42ff-84d8-1bed98edeb60-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894985-jp75m\" (UID: \"be2cbd18-00fc-42ff-84d8-1bed98edeb60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894985-jp75m"
Nov 03 09:45:00.344356 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.344199    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-465707-29894985-wv4z2\" (UID: \"a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894985-wv4z2"
Nov 03 09:45:00.344356 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.344312    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9x66\" (UniqueName: \"kubernetes.io/projected/9ad94c34-df19-4f05-a05d-93b64abe1828-kube-api-access-h9x66\") pod \"maas-api-key-cleanup-e2e-worker-w0-910d50-29894985-fblt9\" (UID: \"9ad94c34-df19-4f05-a05d-93b64abe1828\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894985-fblt9"
Nov 03 09:45:00.344356 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.344335    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4776h\" (UniqueName: \"kubernetes.io/projected/4ffb2aa7-f878-47e7-995e-f5db6f14b5a3-kube-api-access-4776h\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894985-7kqh9\" (UID: \"4ffb2aa7-f878-47e7-995e-f5db6f14b5a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894985-7kqh9"
Nov 03 09:45:00.344472 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.344360    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894985-brvhf\" (UID: \"7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894985-brvhf"
Nov 03 09:45:00.344472 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.344458    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxn7s\" (UniqueName: \"kubernetes.io/projected/a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5-kube-api-access-bxn7s\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-465707-29894985-wv4z2\" (UID: \"a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894985-wv4z2"
Nov 03 09:45:00.344543 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.344475    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79r89\" (UniqueName: \"kubernetes.io/projected/b2433454-f4b1-4109-a28c-5106db36e5ee-kube-api-access-79r89\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894985-cftcl\" (UID: \"b2433454-f4b1-4109-a28c-5106db36e5ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894985-cftcl"
Nov 03 09:45:00.344543 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.344497    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ad94c34-df19-4f05-a05d-93b64abe1828-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-910d50-29894985-fblt9\" (UID: \"9ad94c34-df19-4f05-a05d-93b64abe1828\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894985-fblt9"
Nov 03 09:45:00.344606 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.344552    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjjs6\" (UniqueName: \"kubernetes.io/projected/ea1afde1-03bb-4db5-875f-43845656bf63-kube-api-access-zjjs6\") pod \"maas-api-key-cleanup-e2e-inf-b-af514d-29894985-gzb4m\" (UID: \"ea1afde1-03bb-4db5-875f-43845656bf63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af514d-29894985-gzb4m"
Nov 03 09:45:00.344639 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.344609    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hz8n\" (UniqueName: \"kubernetes.io/projected/e8a01825-bdfa-4a32-889f-f1bc9a93cb79-kube-api-access-2hz8n\") pod \"maas-api-key-cleanup-e2e-worker-w2-77207e-29894985-2j8r6\" (UID: \"e8a01825-bdfa-4a32-889f-f1bc9a93cb79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894985-2j8r6"
Nov 03 09:45:00.344639 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.344626    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsxkg\" (UniqueName: \"kubernetes.io/projected/2423bada-0530-46d9-b873-62f7bd6150c9-kube-api-access-rsxkg\") pod \"maas-api-key-cleanup-e2e-inf-a-872281-29894985-xvvlh\" (UID: \"2423bada-0530-46d9-b873-62f7bd6150c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872281-29894985-xvvlh"
Nov 03 09:45:00.344720 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.344646    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2433454-f4b1-4109-a28c-5106db36e5ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894985-cftcl\" (UID: \"b2433454-f4b1-4109-a28c-5106db36e5ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894985-cftcl"
Nov 03 09:45:00.344720 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.344663    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2423bada-0530-46d9-b873-62f7bd6150c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-872281-29894985-xvvlh\" (UID: \"2423bada-0530-46d9-b873-62f7bd6150c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872281-29894985-xvvlh"
Nov 03 09:45:00.344720 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.344697    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a01825-bdfa-4a32-889f-f1bc9a93cb79-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-77207e-29894985-2j8r6\" (UID: \"e8a01825-bdfa-4a32-889f-f1bc9a93cb79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894985-2j8r6"
Nov 03 09:45:00.344868 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.344834    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd6af3f-fac0-471b-b368-5da14c939450-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894985-9pscs\" (UID: \"1cd6af3f-fac0-471b-b368-5da14c939450\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894985-9pscs"
Nov 03 09:45:00.344927 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.344891    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzv84\" (UniqueName: \"kubernetes.io/projected/be2cbd18-00fc-42ff-84d8-1bed98edeb60-kube-api-access-fzv84\") pod \"maas-api-key-cleanup-29894985-jp75m\" (UID: \"be2cbd18-00fc-42ff-84d8-1bed98edeb60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894985-jp75m"
Nov 03 09:45:00.345042 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.345019    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xj88\" (UniqueName: \"kubernetes.io/projected/7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed-kube-api-access-7xj88\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894985-brvhf\" (UID: \"7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894985-brvhf"
Nov 03 09:45:00.345107 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.345074    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5rcx\" (UniqueName: \"kubernetes.io/projected/1cd6af3f-fac0-471b-b368-5da14c939450-kube-api-access-b5rcx\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894985-9pscs\" (UID: \"1cd6af3f-fac0-471b-b368-5da14c939450\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894985-9pscs"
Nov 03 09:45:00.345150 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.345110    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea1afde1-03bb-4db5-875f-43845656bf63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-af514d-29894985-gzb4m\" (UID: \"ea1afde1-03bb-4db5-875f-43845656bf63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af514d-29894985-gzb4m"
Nov 03 09:45:00.345150 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.345146    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ffb2aa7-f878-47e7-995e-f5db6f14b5a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894985-7kqh9\" (UID: \"4ffb2aa7-f878-47e7-995e-f5db6f14b5a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894985-7kqh9"
Nov 03 09:45:00.345265 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.345233    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dfd9775-ced2-450a-8cc9-f4de56fafa52-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-18f9da-29894985-flfp2\" (UID: \"7dfd9775-ced2-450a-8cc9-f4de56fafa52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894985-flfp2"
Nov 03 09:45:00.345317 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.345283    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-694gf\" (UniqueName: \"kubernetes.io/projected/7dfd9775-ced2-450a-8cc9-f4de56fafa52-kube-api-access-694gf\") pod \"maas-api-key-cleanup-e2e-worker-w4-18f9da-29894985-flfp2\" (UID: \"7dfd9775-ced2-450a-8cc9-f4de56fafa52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894985-flfp2"
Nov 03 09:45:00.381049 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.381015    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894985-flfp2"]
Nov 03 09:45:00.382164 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.382140    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894985-wv4z2"]
Nov 03 09:45:00.386270 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.386244    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af514d-29894985-gzb4m"]
Nov 03 09:45:00.390667 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.390647    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894985-2j8r6"]
Nov 03 09:45:00.395998 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.395964    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894985-fblt9"]
Nov 03 09:45:00.446239 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.446147    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9x66\" (UniqueName: \"kubernetes.io/projected/9ad94c34-df19-4f05-a05d-93b64abe1828-kube-api-access-h9x66\") pod \"maas-api-key-cleanup-e2e-worker-w0-910d50-29894985-fblt9\" (UID: \"9ad94c34-df19-4f05-a05d-93b64abe1828\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894985-fblt9"
Nov 03 09:45:00.446239 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.446187    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4776h\" (UniqueName: \"kubernetes.io/projected/4ffb2aa7-f878-47e7-995e-f5db6f14b5a3-kube-api-access-4776h\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894985-7kqh9\" (UID: \"4ffb2aa7-f878-47e7-995e-f5db6f14b5a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894985-7kqh9"
Nov 03 09:45:00.446239 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.446216    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894985-brvhf\" (UID: \"7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894985-brvhf"
Nov 03 09:45:00.446619 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.446259    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxn7s\" (UniqueName: \"kubernetes.io/projected/a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5-kube-api-access-bxn7s\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-465707-29894985-wv4z2\" (UID: \"a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894985-wv4z2"
Nov 03 09:45:00.446619 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.446285    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79r89\" (UniqueName: \"kubernetes.io/projected/b2433454-f4b1-4109-a28c-5106db36e5ee-kube-api-access-79r89\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894985-cftcl\" (UID: \"b2433454-f4b1-4109-a28c-5106db36e5ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894985-cftcl"
Nov 03 09:45:00.446619 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.446317    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ad94c34-df19-4f05-a05d-93b64abe1828-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-910d50-29894985-fblt9\" (UID: \"9ad94c34-df19-4f05-a05d-93b64abe1828\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894985-fblt9"
Nov 03 09:45:00.446619 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.446356    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjjs6\" (UniqueName: \"kubernetes.io/projected/ea1afde1-03bb-4db5-875f-43845656bf63-kube-api-access-zjjs6\") pod \"maas-api-key-cleanup-e2e-inf-b-af514d-29894985-gzb4m\" (UID: \"ea1afde1-03bb-4db5-875f-43845656bf63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af514d-29894985-gzb4m"
Nov 03 09:45:00.446619 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.446393    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hz8n\" (UniqueName: \"kubernetes.io/projected/e8a01825-bdfa-4a32-889f-f1bc9a93cb79-kube-api-access-2hz8n\") pod \"maas-api-key-cleanup-e2e-worker-w2-77207e-29894985-2j8r6\" (UID: \"e8a01825-bdfa-4a32-889f-f1bc9a93cb79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894985-2j8r6"
Nov 03 09:45:00.446619 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.446420    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsxkg\" (UniqueName: \"kubernetes.io/projected/2423bada-0530-46d9-b873-62f7bd6150c9-kube-api-access-rsxkg\") pod \"maas-api-key-cleanup-e2e-inf-a-872281-29894985-xvvlh\" (UID: \"2423bada-0530-46d9-b873-62f7bd6150c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872281-29894985-xvvlh"
Nov 03 09:45:00.446619 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.446466    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2433454-f4b1-4109-a28c-5106db36e5ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894985-cftcl\" (UID: \"b2433454-f4b1-4109-a28c-5106db36e5ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894985-cftcl"
Nov 03 09:45:00.446619 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.446497    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2423bada-0530-46d9-b873-62f7bd6150c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-872281-29894985-xvvlh\" (UID: \"2423bada-0530-46d9-b873-62f7bd6150c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872281-29894985-xvvlh"
Nov 03 09:45:00.446619 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.446560    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a01825-bdfa-4a32-889f-f1bc9a93cb79-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-77207e-29894985-2j8r6\" (UID: \"e8a01825-bdfa-4a32-889f-f1bc9a93cb79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894985-2j8r6"
Nov 03 09:45:00.446619 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.446606    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd6af3f-fac0-471b-b368-5da14c939450-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894985-9pscs\" (UID: \"1cd6af3f-fac0-471b-b368-5da14c939450\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894985-9pscs"
Nov 03 09:45:00.447213 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.446642    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzv84\" (UniqueName: \"kubernetes.io/projected/be2cbd18-00fc-42ff-84d8-1bed98edeb60-kube-api-access-fzv84\") pod \"maas-api-key-cleanup-29894985-jp75m\" (UID: \"be2cbd18-00fc-42ff-84d8-1bed98edeb60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894985-jp75m"
Nov 03 09:45:00.447213 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.446706    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xj88\" (UniqueName: \"kubernetes.io/projected/7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed-kube-api-access-7xj88\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894985-brvhf\" (UID: \"7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894985-brvhf"
Nov 03 09:45:00.447213 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.446772    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5rcx\" (UniqueName: \"kubernetes.io/projected/1cd6af3f-fac0-471b-b368-5da14c939450-kube-api-access-b5rcx\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894985-9pscs\" (UID: \"1cd6af3f-fac0-471b-b368-5da14c939450\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894985-9pscs"
Nov 03 09:45:00.447213 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.446806    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea1afde1-03bb-4db5-875f-43845656bf63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-af514d-29894985-gzb4m\" (UID: \"ea1afde1-03bb-4db5-875f-43845656bf63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af514d-29894985-gzb4m"
Nov 03 09:45:00.447213 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.446838    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ffb2aa7-f878-47e7-995e-f5db6f14b5a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894985-7kqh9\" (UID: \"4ffb2aa7-f878-47e7-995e-f5db6f14b5a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894985-7kqh9"
Nov 03 09:45:00.447213 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.446887    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dfd9775-ced2-450a-8cc9-f4de56fafa52-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-18f9da-29894985-flfp2\" (UID: \"7dfd9775-ced2-450a-8cc9-f4de56fafa52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894985-flfp2"
Nov 03 09:45:00.447213 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.446963    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-694gf\" (UniqueName: \"kubernetes.io/projected/7dfd9775-ced2-450a-8cc9-f4de56fafa52-kube-api-access-694gf\") pod \"maas-api-key-cleanup-e2e-worker-w4-18f9da-29894985-flfp2\" (UID: \"7dfd9775-ced2-450a-8cc9-f4de56fafa52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894985-flfp2"
Nov 03 09:45:00.447213 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.446999    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be2cbd18-00fc-42ff-84d8-1bed98edeb60-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894985-jp75m\" (UID: \"be2cbd18-00fc-42ff-84d8-1bed98edeb60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894985-jp75m"
Nov 03 09:45:00.447213 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.447023    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-465707-29894985-wv4z2\" (UID: \"a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894985-wv4z2"
Nov 03 09:45:00.447213 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.447164    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2433454-f4b1-4109-a28c-5106db36e5ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894985-cftcl\" (UID: \"b2433454-f4b1-4109-a28c-5106db36e5ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894985-cftcl"
Nov 03 09:45:00.447756 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.447411    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a01825-bdfa-4a32-889f-f1bc9a93cb79-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-77207e-29894985-2j8r6\" (UID: \"e8a01825-bdfa-4a32-889f-f1bc9a93cb79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894985-2j8r6"
Nov 03 09:45:00.447756 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.447442    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ad94c34-df19-4f05-a05d-93b64abe1828-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-910d50-29894985-fblt9\" (UID: \"9ad94c34-df19-4f05-a05d-93b64abe1828\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894985-fblt9"
Nov 03 09:45:00.447756 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.447672    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea1afde1-03bb-4db5-875f-43845656bf63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-af514d-29894985-gzb4m\" (UID: \"ea1afde1-03bb-4db5-875f-43845656bf63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af514d-29894985-gzb4m"
Nov 03 09:45:00.447935 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.447754    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-465707-29894985-wv4z2\" (UID: \"a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894985-wv4z2"
Nov 03 09:45:00.448183 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.448160    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dfd9775-ced2-450a-8cc9-f4de56fafa52-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-18f9da-29894985-flfp2\" (UID: \"7dfd9775-ced2-450a-8cc9-f4de56fafa52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894985-flfp2"
Nov 03 09:45:00.448183 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.448170    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2423bada-0530-46d9-b873-62f7bd6150c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-872281-29894985-xvvlh\" (UID: \"2423bada-0530-46d9-b873-62f7bd6150c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872281-29894985-xvvlh"
Nov 03 09:45:00.448319 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.448180    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd6af3f-fac0-471b-b368-5da14c939450-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894985-9pscs\" (UID: \"1cd6af3f-fac0-471b-b368-5da14c939450\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894985-9pscs"
Nov 03 09:45:00.448723 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.448699    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be2cbd18-00fc-42ff-84d8-1bed98edeb60-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894985-jp75m\" (UID: \"be2cbd18-00fc-42ff-84d8-1bed98edeb60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894985-jp75m"
Nov 03 09:45:00.448811 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.448779    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ffb2aa7-f878-47e7-995e-f5db6f14b5a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894985-7kqh9\" (UID: \"4ffb2aa7-f878-47e7-995e-f5db6f14b5a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894985-7kqh9"
Nov 03 09:45:00.449218 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.449194    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894985-brvhf\" (UID: \"7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894985-brvhf"
Nov 03 09:45:00.510017 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.509984    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9x66\" (UniqueName: \"kubernetes.io/projected/9ad94c34-df19-4f05-a05d-93b64abe1828-kube-api-access-h9x66\") pod \"maas-api-key-cleanup-e2e-worker-w0-910d50-29894985-fblt9\" (UID: \"9ad94c34-df19-4f05-a05d-93b64abe1828\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894985-fblt9"
Nov 03 09:45:00.512002 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.511976    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxn7s\" (UniqueName: \"kubernetes.io/projected/a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5-kube-api-access-bxn7s\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-465707-29894985-wv4z2\" (UID: \"a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894985-wv4z2"
Nov 03 09:45:00.512151 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.511974    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4776h\" (UniqueName: \"kubernetes.io/projected/4ffb2aa7-f878-47e7-995e-f5db6f14b5a3-kube-api-access-4776h\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894985-7kqh9\" (UID: \"4ffb2aa7-f878-47e7-995e-f5db6f14b5a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894985-7kqh9"
Nov 03 09:45:00.512751 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.512729    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hz8n\" (UniqueName: \"kubernetes.io/projected/e8a01825-bdfa-4a32-889f-f1bc9a93cb79-kube-api-access-2hz8n\") pod \"maas-api-key-cleanup-e2e-worker-w2-77207e-29894985-2j8r6\" (UID: \"e8a01825-bdfa-4a32-889f-f1bc9a93cb79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894985-2j8r6"
Nov 03 09:45:00.514228 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.514201    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjjs6\" (UniqueName: \"kubernetes.io/projected/ea1afde1-03bb-4db5-875f-43845656bf63-kube-api-access-zjjs6\") pod \"maas-api-key-cleanup-e2e-inf-b-af514d-29894985-gzb4m\" (UID: \"ea1afde1-03bb-4db5-875f-43845656bf63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af514d-29894985-gzb4m"
Nov 03 09:45:00.514420 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.514395    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsxkg\" (UniqueName: \"kubernetes.io/projected/2423bada-0530-46d9-b873-62f7bd6150c9-kube-api-access-rsxkg\") pod \"maas-api-key-cleanup-e2e-inf-a-872281-29894985-xvvlh\" (UID: \"2423bada-0530-46d9-b873-62f7bd6150c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872281-29894985-xvvlh"
Nov 03 09:45:00.514494 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.514461    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5rcx\" (UniqueName: \"kubernetes.io/projected/1cd6af3f-fac0-471b-b368-5da14c939450-kube-api-access-b5rcx\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894985-9pscs\" (UID: \"1cd6af3f-fac0-471b-b368-5da14c939450\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894985-9pscs"
Nov 03 09:45:00.516887 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.516865    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-694gf\" (UniqueName: \"kubernetes.io/projected/7dfd9775-ced2-450a-8cc9-f4de56fafa52-kube-api-access-694gf\") pod \"maas-api-key-cleanup-e2e-worker-w4-18f9da-29894985-flfp2\" (UID: \"7dfd9775-ced2-450a-8cc9-f4de56fafa52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894985-flfp2"
Nov 03 09:45:00.517013 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.516994    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79r89\" (UniqueName: \"kubernetes.io/projected/b2433454-f4b1-4109-a28c-5106db36e5ee-kube-api-access-79r89\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894985-cftcl\" (UID: \"b2433454-f4b1-4109-a28c-5106db36e5ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894985-cftcl"
Nov 03 09:45:00.518330 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.518312    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzv84\" (UniqueName: \"kubernetes.io/projected/be2cbd18-00fc-42ff-84d8-1bed98edeb60-kube-api-access-fzv84\") pod \"maas-api-key-cleanup-29894985-jp75m\" (UID: \"be2cbd18-00fc-42ff-84d8-1bed98edeb60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894985-jp75m"
Nov 03 09:45:00.518846 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.518830    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xj88\" (UniqueName: \"kubernetes.io/projected/7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed-kube-api-access-7xj88\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894985-brvhf\" (UID: \"7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894985-brvhf"
Nov 03 09:45:00.533341 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.533317    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894985-cftcl"
Nov 03 09:45:00.543904 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.543882    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872281-29894985-xvvlh"
Nov 03 09:45:00.551196 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.551172    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894985-9pscs"
Nov 03 09:45:00.559779 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.559756    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894985-7kqh9"
Nov 03 09:45:00.573086 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.568804    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894985-wv4z2"
Nov 03 09:45:00.578461 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.578437    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894985-fblt9"
Nov 03 09:45:00.588738 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.588616    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894985-2j8r6"
Nov 03 09:45:00.592281 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.592113    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894985-flfp2"
Nov 03 09:45:00.786245 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.785995    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894985-brvhf"
Nov 03 09:45:00.793099 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.793074    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894985-jp75m"
Nov 03 09:45:00.798986 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.798960    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af514d-29894985-gzb4m"
Nov 03 09:45:00.938068 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.938038    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894985-cftcl"]
Nov 03 09:45:00.939332 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:45:00.939300    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2433454_f4b1_4109_a28c_5106db36e5ee.slice/crio-cfc6feb96516f9e9a671c888d30d2c03ad76e54384c70f0977f4d8733d564ba8 WatchSource:0}: Error finding container cfc6feb96516f9e9a671c888d30d2c03ad76e54384c70f0977f4d8733d564ba8: Status 404 returned error can't find the container with id cfc6feb96516f9e9a671c888d30d2c03ad76e54384c70f0977f4d8733d564ba8
Nov 03 09:45:00.989042 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.989015    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872281-29894985-xvvlh"]
Nov 03 09:45:00.990456 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:00.990437    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894985-9pscs"]
Nov 03 09:45:01.008891 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:45:01.007585    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2423bada_0530_46d9_b873_62f7bd6150c9.slice/crio-4328045c49c09ab4dd86ef7b95131ead596a946c40b2f93c4d34c83dac9745ee WatchSource:0}: Error finding container 4328045c49c09ab4dd86ef7b95131ead596a946c40b2f93c4d34c83dac9745ee: Status 404 returned error can't find the container with id 4328045c49c09ab4dd86ef7b95131ead596a946c40b2f93c4d34c83dac9745ee
Nov 03 09:45:01.008891 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:45:01.007846    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd6af3f_fac0_471b_b368_5da14c939450.slice/crio-cefaa69c90fdabe10c8e1dde0868d2d896686be80c356b0c939bdc845f5709ab WatchSource:0}: Error finding container cefaa69c90fdabe10c8e1dde0868d2d896686be80c356b0c939bdc845f5709ab: Status 404 returned error can't find the container with id cefaa69c90fdabe10c8e1dde0868d2d896686be80c356b0c939bdc845f5709ab
Nov 03 09:45:01.095029 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:01.095002    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894985-2j8r6"]
Nov 03 09:45:01.118888 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:45:01.118852    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a01825_bdfa_4a32_889f_f1bc9a93cb79.slice/crio-c718e13d736b5218780dfb2328cb1aa5284f5896cb6d16f6de2322eec0ba11f0 WatchSource:0}: Error finding container c718e13d736b5218780dfb2328cb1aa5284f5896cb6d16f6de2322eec0ba11f0: Status 404 returned error can't find the container with id c718e13d736b5218780dfb2328cb1aa5284f5896cb6d16f6de2322eec0ba11f0
Nov 03 09:45:01.141072 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:45:01.141040    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dfd9775_ced2_450a_8cc9_f4de56fafa52.slice/crio-c6d5a25e5079a484268d45456bfa2df3206b50ec3e13e3336049f290a9443443 WatchSource:0}: Error finding container c6d5a25e5079a484268d45456bfa2df3206b50ec3e13e3336049f290a9443443: Status 404 returned error can't find the container with id c6d5a25e5079a484268d45456bfa2df3206b50ec3e13e3336049f290a9443443
Nov 03 09:45:01.141332 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:01.141311    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894985-flfp2"]
Nov 03 09:45:01.204886 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:01.204856    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894985-brvhf"]
Nov 03 09:45:01.205788 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:45:01.205756    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e39b5ea_acdb_4a9e_bc1b_bdf9bd4553ed.slice/crio-42ed0c988557a4bfb433abb19c4fa7c4f4f3edc9781b1f94725d0909d5e39c22 WatchSource:0}: Error finding container 42ed0c988557a4bfb433abb19c4fa7c4f4f3edc9781b1f94725d0909d5e39c22: Status 404 returned error can't find the container with id 42ed0c988557a4bfb433abb19c4fa7c4f4f3edc9781b1f94725d0909d5e39c22
Nov 03 09:45:01.232892 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:01.232870    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894985-jp75m"]
Nov 03 09:45:01.253212 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:01.253188    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894985-7kqh9"]
Nov 03 09:45:01.262028 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:01.262002    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af514d-29894985-gzb4m"]
Nov 03 09:45:01.293777 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:01.293748    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894985-fblt9"]
Nov 03 09:45:01.321507 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:45:01.321478    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe2cbd18_00fc_42ff_84d8_1bed98edeb60.slice/crio-9ba058b339451e69dee1363206e81fde8a4992397c1717189cd0d0193cc6bea6 WatchSource:0}: Error finding container 9ba058b339451e69dee1363206e81fde8a4992397c1717189cd0d0193cc6bea6: Status 404 returned error can't find the container with id 9ba058b339451e69dee1363206e81fde8a4992397c1717189cd0d0193cc6bea6
Nov 03 09:45:01.326039 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:45:01.326007    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ffb2aa7_f878_47e7_995e_f5db6f14b5a3.slice/crio-e9196d40c0e24e34203c625b24027d53c80cae1375db181cd2683cef4e8bda2f WatchSource:0}: Error finding container e9196d40c0e24e34203c625b24027d53c80cae1375db181cd2683cef4e8bda2f: Status 404 returned error can't find the container with id e9196d40c0e24e34203c625b24027d53c80cae1375db181cd2683cef4e8bda2f
Nov 03 09:45:01.342180 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:01.342145    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894985-wv4z2"]
Nov 03 09:45:01.357354 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:45:01.357319    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4730ab7_f5a6_4e3f_9ad8_ba020c5a48a5.slice/crio-4767af0fb30392ebdb114db21e59fa1e5b0fa6c644b4faf567b29859bdc29088 WatchSource:0}: Error finding container 4767af0fb30392ebdb114db21e59fa1e5b0fa6c644b4faf567b29859bdc29088: Status 404 returned error can't find the container with id 4767af0fb30392ebdb114db21e59fa1e5b0fa6c644b4faf567b29859bdc29088
Nov 03 09:45:01.503759 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:01.503670    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872281-29894985-xvvlh"]
Nov 03 09:45:01.884765 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:01.884683    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894985-fblt9" event={"ID":"9ad94c34-df19-4f05-a05d-93b64abe1828","Type":"ContainerStarted","Data":"0a480a2742389c32a28853fa4ad2b0d92e1175d840f5541267b618471a8fa413"}
Nov 03 09:45:01.889547 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:01.889519    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872281-29894985-xvvlh" event={"ID":"2423bada-0530-46d9-b873-62f7bd6150c9","Type":"ContainerStarted","Data":"832d3b22c3cb241fc5dad8e9f84eb24fced6f3da7defd3444ac301e0a85e39fd"}
Nov 03 09:45:01.889667 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:01.889598    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872281-29894985-xvvlh" event={"ID":"2423bada-0530-46d9-b873-62f7bd6150c9","Type":"ContainerStarted","Data":"4328045c49c09ab4dd86ef7b95131ead596a946c40b2f93c4d34c83dac9745ee"}
Nov 03 09:45:01.889667 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:01.889635    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872281-29894985-xvvlh" podUID="2423bada-0530-46d9-b873-62f7bd6150c9" containerName="cleanup" containerID="cri-o://832d3b22c3cb241fc5dad8e9f84eb24fced6f3da7defd3444ac301e0a85e39fd" gracePeriod=30
Nov 03 09:45:01.893837 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:01.893081    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894985-9pscs" event={"ID":"1cd6af3f-fac0-471b-b368-5da14c939450","Type":"ContainerStarted","Data":"864ef5a7ef9b707cd1b269b9f5a96a3de1e3629daaaac5ba81dc8eeb4d04e41a"}
Nov 03 09:45:01.893837 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:01.893138    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894985-9pscs" event={"ID":"1cd6af3f-fac0-471b-b368-5da14c939450","Type":"ContainerStarted","Data":"cefaa69c90fdabe10c8e1dde0868d2d896686be80c356b0c939bdc845f5709ab"}
Nov 03 09:45:01.910063 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:01.909217    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894985-cftcl" event={"ID":"b2433454-f4b1-4109-a28c-5106db36e5ee","Type":"ContainerStarted","Data":"fa8d0a21d7f245d17e9d86bbea135175d0bb18dbe2013319c9e49c807e415bef"}
Nov 03 09:45:01.910063 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:01.909305    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894985-cftcl" event={"ID":"b2433454-f4b1-4109-a28c-5106db36e5ee","Type":"ContainerStarted","Data":"cfc6feb96516f9e9a671c888d30d2c03ad76e54384c70f0977f4d8733d564ba8"}
Nov 03 09:45:01.913573 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:01.912680    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894985-brvhf" event={"ID":"7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed","Type":"ContainerStarted","Data":"42ed0c988557a4bfb433abb19c4fa7c4f4f3edc9781b1f94725d0909d5e39c22"}
Nov 03 09:45:01.915682 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:01.915652    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af514d-29894985-gzb4m" event={"ID":"ea1afde1-03bb-4db5-875f-43845656bf63","Type":"ContainerStarted","Data":"f39dc76bd456a706240d41a7646606e1e998bf12f2dddd6f73a32758161959e9"}
Nov 03 09:45:01.951987 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:01.946291    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894985-flfp2" event={"ID":"7dfd9775-ced2-450a-8cc9-f4de56fafa52","Type":"ContainerStarted","Data":"5769528b43b2229e5e325f9649eebbcfda03d64acc39e6aa66fab68696cc602f"}
Nov 03 09:45:01.951987 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:01.946353    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894985-flfp2" event={"ID":"7dfd9775-ced2-450a-8cc9-f4de56fafa52","Type":"ContainerStarted","Data":"c6d5a25e5079a484268d45456bfa2df3206b50ec3e13e3336049f290a9443443"}
Nov 03 09:45:01.952356 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:01.952292    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894985-wv4z2" event={"ID":"a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5","Type":"ContainerStarted","Data":"4767af0fb30392ebdb114db21e59fa1e5b0fa6c644b4faf567b29859bdc29088"}
Nov 03 09:45:01.958773 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:01.958286    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894985-jp75m" event={"ID":"be2cbd18-00fc-42ff-84d8-1bed98edeb60","Type":"ContainerStarted","Data":"9ba058b339451e69dee1363206e81fde8a4992397c1717189cd0d0193cc6bea6"}
Nov 03 09:45:01.961533 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:01.961423    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894985-7kqh9" event={"ID":"4ffb2aa7-f878-47e7-995e-f5db6f14b5a3","Type":"ContainerStarted","Data":"e9196d40c0e24e34203c625b24027d53c80cae1375db181cd2683cef4e8bda2f"}
Nov 03 09:45:01.963759 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:01.963735    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894985-2j8r6" event={"ID":"e8a01825-bdfa-4a32-889f-f1bc9a93cb79","Type":"ContainerStarted","Data":"532f4df5a8ff1c486878ee9c29e63b79019e7f0f37fc935842c0f111acd1874d"}
Nov 03 09:45:01.963898 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:01.963881    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894985-2j8r6" event={"ID":"e8a01825-bdfa-4a32-889f-f1bc9a93cb79","Type":"ContainerStarted","Data":"c718e13d736b5218780dfb2328cb1aa5284f5896cb6d16f6de2322eec0ba11f0"}
Nov 03 09:45:02.006308 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.006265    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872281-29894985-xvvlh" podStartSLOduration=2.006251576 podStartE2EDuration="2.006251576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:45:02.005264951 +0000 UTC m=+3926.274328403" watchObservedRunningTime="2026-11-03 09:45:02.006251576 +0000 UTC m=+3926.275315001"
Nov 03 09:45:02.238501 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.238447    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894985-7kqh9" podStartSLOduration=2.238432277 podStartE2EDuration="2.238432277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:45:02.121218064 +0000 UTC m=+3926.390281498" watchObservedRunningTime="2026-11-03 09:45:02.238432277 +0000 UTC m=+3926.507495711"
Nov 03 09:45:02.239508 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.239468    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894985-flfp2" podStartSLOduration=2.239457207 podStartE2EDuration="2.239457207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:45:02.236363984 +0000 UTC m=+3926.505427428" watchObservedRunningTime="2026-11-03 09:45:02.239457207 +0000 UTC m=+3926.508520642"
Nov 03 09:45:02.363018 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.362744    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894985-brvhf" podStartSLOduration=2.362724997 podStartE2EDuration="2.362724997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:45:02.362418527 +0000 UTC m=+3926.631481993" watchObservedRunningTime="2026-11-03 09:45:02.362724997 +0000 UTC m=+3926.631788432"
Nov 03 09:45:02.482705 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.482654    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894985-2j8r6" podStartSLOduration=2.482639501 podStartE2EDuration="2.482639501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:45:02.479002083 +0000 UTC m=+3926.748065540" watchObservedRunningTime="2026-11-03 09:45:02.482639501 +0000 UTC m=+3926.751702935"
Nov 03 09:45:02.601065 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.601018    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894985-9pscs" podStartSLOduration=2.601002534 podStartE2EDuration="2.601002534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:45:02.596787431 +0000 UTC m=+3926.865850886" watchObservedRunningTime="2026-11-03 09:45:02.601002534 +0000 UTC m=+3926.870065998"
Nov 03 09:45:02.712397 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.712348    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894985-wv4z2" podStartSLOduration=2.7123371499999998 podStartE2EDuration="2.71233715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:45:02.710416883 +0000 UTC m=+3926.979480341" watchObservedRunningTime="2026-11-03 09:45:02.71233715 +0000 UTC m=+3926.981400585"
Nov 03 09:45:02.826245 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.826190    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894985-cftcl" podStartSLOduration=2.826172711 podStartE2EDuration="2.826172711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:45:02.825568303 +0000 UTC m=+3927.094631736" watchObservedRunningTime="2026-11-03 09:45:02.826172711 +0000 UTC m=+3927.095236142"
Nov 03 09:45:02.856215 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.856186    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872281-29894985-xvvlh"
Nov 03 09:45:02.941901 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.941849    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894985-jp75m" podStartSLOduration=2.941834714 podStartE2EDuration="2.941834714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:45:02.939807348 +0000 UTC m=+3927.208870797" watchObservedRunningTime="2026-11-03 09:45:02.941834714 +0000 UTC m=+3927.210898148"
Nov 03 09:45:02.971938 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.971900    4473 generic.go:356] "Generic (PLEG): container finished" podID="ea1afde1-03bb-4db5-875f-43845656bf63" containerID="2ba25c2e9a30f41dab51025e9526eee45139e8a64deca8b2546650f658e10281" exitCode=60
Nov 03 09:45:02.972146 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.971997    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af514d-29894985-gzb4m" event={"ID":"ea1afde1-03bb-4db5-875f-43845656bf63","Type":"ContainerDied","Data":"2ba25c2e9a30f41dab51025e9526eee45139e8a64deca8b2546650f658e10281"}
Nov 03 09:45:02.972376 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.972356    4473 scope.go:122] "RemoveContainer" containerID="2ba25c2e9a30f41dab51025e9526eee45139e8a64deca8b2546650f658e10281"
Nov 03 09:45:02.974043 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.973932    4473 generic.go:356] "Generic (PLEG): container finished" podID="7dfd9775-ced2-450a-8cc9-f4de56fafa52" containerID="5769528b43b2229e5e325f9649eebbcfda03d64acc39e6aa66fab68696cc602f" exitCode=60
Nov 03 09:45:02.974043 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.973989    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894985-flfp2" event={"ID":"7dfd9775-ced2-450a-8cc9-f4de56fafa52","Type":"ContainerDied","Data":"5769528b43b2229e5e325f9649eebbcfda03d64acc39e6aa66fab68696cc602f"}
Nov 03 09:45:02.974318 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.974297    4473 scope.go:122] "RemoveContainer" containerID="5769528b43b2229e5e325f9649eebbcfda03d64acc39e6aa66fab68696cc602f"
Nov 03 09:45:02.975902 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.975810    4473 generic.go:356] "Generic (PLEG): container finished" podID="a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5" containerID="f00a124843dd34c58d5818dc5627c52fd5cb7ed749cabf9456a756e7b6cc6347" exitCode=60
Nov 03 09:45:02.975902 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.975838    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894985-wv4z2" event={"ID":"a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5","Type":"ContainerDied","Data":"f00a124843dd34c58d5818dc5627c52fd5cb7ed749cabf9456a756e7b6cc6347"}
Nov 03 09:45:02.976172 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.976153    4473 scope.go:122] "RemoveContainer" containerID="f00a124843dd34c58d5818dc5627c52fd5cb7ed749cabf9456a756e7b6cc6347"
Nov 03 09:45:02.977644 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.977572    4473 generic.go:356] "Generic (PLEG): container finished" podID="be2cbd18-00fc-42ff-84d8-1bed98edeb60" containerID="8f3f5bcfe3231a73adf232382c3ec02ec8a6c950516f98cf16b29a998e5f2e59" exitCode=60
Nov 03 09:45:02.977644 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.977621    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894985-jp75m" event={"ID":"be2cbd18-00fc-42ff-84d8-1bed98edeb60","Type":"ContainerDied","Data":"8f3f5bcfe3231a73adf232382c3ec02ec8a6c950516f98cf16b29a998e5f2e59"}
Nov 03 09:45:02.977843 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.977827    4473 scope.go:122] "RemoveContainer" containerID="8f3f5bcfe3231a73adf232382c3ec02ec8a6c950516f98cf16b29a998e5f2e59"
Nov 03 09:45:02.979182 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.979164    4473 generic.go:356] "Generic (PLEG): container finished" podID="4ffb2aa7-f878-47e7-995e-f5db6f14b5a3" containerID="e262adaeb6a6ec0b65c56b74bd8740ac780b7e9103707c0907bcda2d24c359dd" exitCode=60
Nov 03 09:45:02.979283 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.979232    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894985-7kqh9" event={"ID":"4ffb2aa7-f878-47e7-995e-f5db6f14b5a3","Type":"ContainerDied","Data":"e262adaeb6a6ec0b65c56b74bd8740ac780b7e9103707c0907bcda2d24c359dd"}
Nov 03 09:45:02.979432 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.979416    4473 scope.go:122] "RemoveContainer" containerID="e262adaeb6a6ec0b65c56b74bd8740ac780b7e9103707c0907bcda2d24c359dd"
Nov 03 09:45:02.980862 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.980829    4473 generic.go:356] "Generic (PLEG): container finished" podID="e8a01825-bdfa-4a32-889f-f1bc9a93cb79" containerID="532f4df5a8ff1c486878ee9c29e63b79019e7f0f37fc935842c0f111acd1874d" exitCode=60
Nov 03 09:45:02.980978 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.980913    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894985-2j8r6" event={"ID":"e8a01825-bdfa-4a32-889f-f1bc9a93cb79","Type":"ContainerDied","Data":"532f4df5a8ff1c486878ee9c29e63b79019e7f0f37fc935842c0f111acd1874d"}
Nov 03 09:45:02.981143 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.981122    4473 scope.go:122] "RemoveContainer" containerID="532f4df5a8ff1c486878ee9c29e63b79019e7f0f37fc935842c0f111acd1874d"
Nov 03 09:45:02.982868 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.982846    4473 generic.go:356] "Generic (PLEG): container finished" podID="9ad94c34-df19-4f05-a05d-93b64abe1828" containerID="bbcc6bef4c8dcccc9affb6e285d8f71b3e82e107dbc2843cf8603d463a13739f" exitCode=60
Nov 03 09:45:02.982995 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.982926    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894985-fblt9" event={"ID":"9ad94c34-df19-4f05-a05d-93b64abe1828","Type":"ContainerDied","Data":"bbcc6bef4c8dcccc9affb6e285d8f71b3e82e107dbc2843cf8603d463a13739f"}
Nov 03 09:45:02.983229 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.983209    4473 scope.go:122] "RemoveContainer" containerID="bbcc6bef4c8dcccc9affb6e285d8f71b3e82e107dbc2843cf8603d463a13739f"
Nov 03 09:45:02.984983 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.984962    4473 generic.go:356] "Generic (PLEG): container finished" podID="2423bada-0530-46d9-b873-62f7bd6150c9" containerID="832d3b22c3cb241fc5dad8e9f84eb24fced6f3da7defd3444ac301e0a85e39fd" exitCode=6
Nov 03 09:45:02.985056 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.984997    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872281-29894985-xvvlh"
Nov 03 09:45:02.985056 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.985043    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872281-29894985-xvvlh" event={"ID":"2423bada-0530-46d9-b873-62f7bd6150c9","Type":"ContainerDied","Data":"832d3b22c3cb241fc5dad8e9f84eb24fced6f3da7defd3444ac301e0a85e39fd"}
Nov 03 09:45:02.985167 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.985097    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872281-29894985-xvvlh" event={"ID":"2423bada-0530-46d9-b873-62f7bd6150c9","Type":"ContainerDied","Data":"4328045c49c09ab4dd86ef7b95131ead596a946c40b2f93c4d34c83dac9745ee"}
Nov 03 09:45:02.985167 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.985125    4473 scope.go:122] "RemoveContainer" containerID="832d3b22c3cb241fc5dad8e9f84eb24fced6f3da7defd3444ac301e0a85e39fd"
Nov 03 09:45:02.986343 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.986322    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2423bada-0530-46d9-b873-62f7bd6150c9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2423bada-0530-46d9-b873-62f7bd6150c9-maas-api-service-ca\") pod \"2423bada-0530-46d9-b873-62f7bd6150c9\" (UID: \"2423bada-0530-46d9-b873-62f7bd6150c9\") "
Nov 03 09:45:02.986435 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.986396    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2423bada-0530-46d9-b873-62f7bd6150c9-kube-api-access-rsxkg\" (UniqueName: \"kubernetes.io/projected/2423bada-0530-46d9-b873-62f7bd6150c9-kube-api-access-rsxkg\") pod \"2423bada-0530-46d9-b873-62f7bd6150c9\" (UID: \"2423bada-0530-46d9-b873-62f7bd6150c9\") "
Nov 03 09:45:02.986720 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.986689    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2423bada-0530-46d9-b873-62f7bd6150c9-maas-api-service-ca" pod "2423bada-0530-46d9-b873-62f7bd6150c9" (UID: "2423bada-0530-46d9-b873-62f7bd6150c9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:45:02.986836 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.986817    4473 generic.go:356] "Generic (PLEG): container finished" podID="1cd6af3f-fac0-471b-b368-5da14c939450" containerID="864ef5a7ef9b707cd1b269b9f5a96a3de1e3629daaaac5ba81dc8eeb4d04e41a" exitCode=60
Nov 03 09:45:02.987673 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.987085    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2423bada-0530-46d9-b873-62f7bd6150c9-maas-api-service-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:45:02.987673 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.987307    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894985-9pscs" event={"ID":"1cd6af3f-fac0-471b-b368-5da14c939450","Type":"ContainerDied","Data":"864ef5a7ef9b707cd1b269b9f5a96a3de1e3629daaaac5ba81dc8eeb4d04e41a"}
Nov 03 09:45:02.988182 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.988167    4473 scope.go:122] "RemoveContainer" containerID="864ef5a7ef9b707cd1b269b9f5a96a3de1e3629daaaac5ba81dc8eeb4d04e41a"
Nov 03 09:45:02.988748 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.988731    4473 generic.go:356] "Generic (PLEG): container finished" podID="b2433454-f4b1-4109-a28c-5106db36e5ee" containerID="fa8d0a21d7f245d17e9d86bbea135175d0bb18dbe2013319c9e49c807e415bef" exitCode=60
Nov 03 09:45:02.988839 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.988826    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894985-cftcl" event={"ID":"b2433454-f4b1-4109-a28c-5106db36e5ee","Type":"ContainerDied","Data":"fa8d0a21d7f245d17e9d86bbea135175d0bb18dbe2013319c9e49c807e415bef"}
Nov 03 09:45:02.989115 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.989094    4473 scope.go:122] "RemoveContainer" containerID="fa8d0a21d7f245d17e9d86bbea135175d0bb18dbe2013319c9e49c807e415bef"
Nov 03 09:45:02.989253 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.989227    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2423bada-0530-46d9-b873-62f7bd6150c9-kube-api-access-rsxkg" pod "2423bada-0530-46d9-b873-62f7bd6150c9" (UID: "2423bada-0530-46d9-b873-62f7bd6150c9"). InnerVolumeSpecName "kube-api-access-rsxkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:45:02.991309 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.991259    4473 generic.go:356] "Generic (PLEG): container finished" podID="7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed" containerID="d361220f359bf8afa1fb6b48b37648f3e07fde640680c871d06eb3c2775ce84b" exitCode=60
Nov 03 09:45:02.991374 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.991331    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894985-brvhf" event={"ID":"7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed","Type":"ContainerDied","Data":"d361220f359bf8afa1fb6b48b37648f3e07fde640680c871d06eb3c2775ce84b"}
Nov 03 09:45:02.991607 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:02.991593    4473 scope.go:122] "RemoveContainer" containerID="d361220f359bf8afa1fb6b48b37648f3e07fde640680c871d06eb3c2775ce84b"
Nov 03 09:45:03.044361 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:03.044333    4473 scope.go:122] "RemoveContainer" containerID="832d3b22c3cb241fc5dad8e9f84eb24fced6f3da7defd3444ac301e0a85e39fd"
Nov 03 09:45:03.044816 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:03.044786    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"832d3b22c3cb241fc5dad8e9f84eb24fced6f3da7defd3444ac301e0a85e39fd\": container with ID starting with 832d3b22c3cb241fc5dad8e9f84eb24fced6f3da7defd3444ac301e0a85e39fd not found: ID does not exist" containerID="832d3b22c3cb241fc5dad8e9f84eb24fced6f3da7defd3444ac301e0a85e39fd"
Nov 03 09:45:03.044920 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:03.044826    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"832d3b22c3cb241fc5dad8e9f84eb24fced6f3da7defd3444ac301e0a85e39fd"} err="failed to get container status \"832d3b22c3cb241fc5dad8e9f84eb24fced6f3da7defd3444ac301e0a85e39fd\": rpc error: code = NotFound desc = could not find container \"832d3b22c3cb241fc5dad8e9f84eb24fced6f3da7defd3444ac301e0a85e39fd\": container with ID starting with 832d3b22c3cb241fc5dad8e9f84eb24fced6f3da7defd3444ac301e0a85e39fd not found: ID does not exist"
Nov 03 09:45:03.068023 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:03.067982    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af514d-29894985-gzb4m" podStartSLOduration=3.067967793 podStartE2EDuration="3.067967793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:45:03.05681358 +0000 UTC m=+3927.325877015" watchObservedRunningTime="2026-11-03 09:45:03.067967793 +0000 UTC m=+3927.337031223"
Nov 03 09:45:03.069887 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:03.069865    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-75cc44f8d5-hqht4"]
Nov 03 09:45:03.070563 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:03.070547    4473 cpu_manager.go:414] "containerMap: removing container" podUID="2423bada-0530-46d9-b873-62f7bd6150c9" containerName="cleanup"
Nov 03 09:45:03.070616 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:03.070566    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2423bada-0530-46d9-b873-62f7bd6150c9" containerName="cleanup"
Nov 03 09:45:03.070697 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:03.070688    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="2423bada-0530-46d9-b873-62f7bd6150c9" containerName="cleanup"
Nov 03 09:45:03.075811 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:03.075790    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75cc44f8d5-hqht4"
Nov 03 09:45:03.089620 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:03.089591    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsxkg\" (UniqueName: \"kubernetes.io/projected/2423bada-0530-46d9-b873-62f7bd6150c9-kube-api-access-rsxkg\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:45:03.130340 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:03.130316    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-b2395a48f4ab\""
Nov 03 09:45:03.131068 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:03.131047    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-75cc44f8d5-hqht4"]
Nov 03 09:45:03.190259 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:03.190223    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59bdg\" (UniqueName: \"kubernetes.io/projected/35553ad3-951a-4f65-ba00-559a9175bf40-kube-api-access-59bdg\") pod \"payload-processing-external-model-75cc44f8d5-hqht4\" (UID: \"35553ad3-951a-4f65-ba00-559a9175bf40\") " pod="models-as-a-service/payload-processing-external-model-75cc44f8d5-hqht4"
Nov 03 09:45:03.190259 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:03.190262    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35553ad3-951a-4f65-ba00-559a9175bf40-routing-overlay-volume\") pod \"payload-processing-external-model-75cc44f8d5-hqht4\" (UID: \"35553ad3-951a-4f65-ba00-559a9175bf40\") " pod="models-as-a-service/payload-processing-external-model-75cc44f8d5-hqht4"
Nov 03 09:45:03.190473 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:03.190425    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35553ad3-951a-4f65-ba00-559a9175bf40-plugins-config-volume\") pod \"payload-processing-external-model-75cc44f8d5-hqht4\" (UID: \"35553ad3-951a-4f65-ba00-559a9175bf40\") " pod="models-as-a-service/payload-processing-external-model-75cc44f8d5-hqht4"
Nov 03 09:45:03.190473 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:03.190447    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/35553ad3-951a-4f65-ba00-559a9175bf40-provider-credentials\") pod \"payload-processing-external-model-75cc44f8d5-hqht4\" (UID: \"35553ad3-951a-4f65-ba00-559a9175bf40\") " pod="models-as-a-service/payload-processing-external-model-75cc44f8d5-hqht4"
Nov 03 09:45:03.320013 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:03.314867    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59bdg\" (UniqueName: \"kubernetes.io/projected/35553ad3-951a-4f65-ba00-559a9175bf40-kube-api-access-59bdg\") pod \"payload-processing-external-model-75cc44f8d5-hqht4\" (UID: \"35553ad3-951a-4f65-ba00-559a9175bf40\") " pod="models-as-a-service/payload-processing-external-model-75cc44f8d5-hqht4"
Nov 03 09:45:03.320013 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:03.314926    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35553ad3-951a-4f65-ba00-559a9175bf40-routing-overlay-volume\") pod \"payload-processing-external-model-75cc44f8d5-hqht4\" (UID: \"35553ad3-951a-4f65-ba00-559a9175bf40\") " pod="models-as-a-service/payload-processing-external-model-75cc44f8d5-hqht4"
Nov 03 09:45:03.320013 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:03.315072    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35553ad3-951a-4f65-ba00-559a9175bf40-plugins-config-volume\") pod \"payload-processing-external-model-75cc44f8d5-hqht4\" (UID: \"35553ad3-951a-4f65-ba00-559a9175bf40\") " pod="models-as-a-service/payload-processing-external-model-75cc44f8d5-hqht4"
Nov 03 09:45:03.320013 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:03.315103    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/35553ad3-951a-4f65-ba00-559a9175bf40-provider-credentials\") pod \"payload-processing-external-model-75cc44f8d5-hqht4\" (UID: \"35553ad3-951a-4f65-ba00-559a9175bf40\") " pod="models-as-a-service/payload-processing-external-model-75cc44f8d5-hqht4"
Nov 03 09:45:03.320013 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:03.315828    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35553ad3-951a-4f65-ba00-559a9175bf40-routing-overlay-volume\") pod \"payload-processing-external-model-75cc44f8d5-hqht4\" (UID: \"35553ad3-951a-4f65-ba00-559a9175bf40\") " pod="models-as-a-service/payload-processing-external-model-75cc44f8d5-hqht4"
Nov 03 09:45:03.320013 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:03.316034    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35553ad3-951a-4f65-ba00-559a9175bf40-plugins-config-volume\") pod \"payload-processing-external-model-75cc44f8d5-hqht4\" (UID: \"35553ad3-951a-4f65-ba00-559a9175bf40\") " pod="models-as-a-service/payload-processing-external-model-75cc44f8d5-hqht4"
Nov 03 09:45:03.334963 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:03.334912    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/35553ad3-951a-4f65-ba00-559a9175bf40-provider-credentials\") pod \"payload-processing-external-model-75cc44f8d5-hqht4\" (UID: \"35553ad3-951a-4f65-ba00-559a9175bf40\") " pod="models-as-a-service/payload-processing-external-model-75cc44f8d5-hqht4"
Nov 03 09:45:03.378448 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:03.378394    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59bdg\" (UniqueName: \"kubernetes.io/projected/35553ad3-951a-4f65-ba00-559a9175bf40-kube-api-access-59bdg\") pod \"payload-processing-external-model-75cc44f8d5-hqht4\" (UID: \"35553ad3-951a-4f65-ba00-559a9175bf40\") " pod="models-as-a-service/payload-processing-external-model-75cc44f8d5-hqht4"
Nov 03 09:45:03.398796 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:03.398759    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75cc44f8d5-hqht4"
Nov 03 09:45:03.967815 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:03.967759    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-75cc44f8d5-hqht4"]
Nov 03 09:45:04.007658 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:04.007628    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75cc44f8d5-hqht4" event={"ID":"35553ad3-951a-4f65-ba00-559a9175bf40","Type":"ContainerStarted","Data":"7178c37e83d6109e7c36bb1b1a6e73940dfb39903cdae36e32052a67cdd7cf55"}
Nov 03 09:45:04.009571 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:04.009543    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af514d-29894985-gzb4m" event={"ID":"ea1afde1-03bb-4db5-875f-43845656bf63","Type":"ContainerStarted","Data":"b6be289ac64a255263ef4f45b1dcb303e9179e1ac0fad09eaf54f24d977bf4e7"}
Nov 03 09:45:04.011372 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:04.011350    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894985-flfp2" event={"ID":"7dfd9775-ced2-450a-8cc9-f4de56fafa52","Type":"ContainerStarted","Data":"fe53f101ff034f0e489aed9d9e0bb494adb722d793d7e9a2fb63f129eec83b58"}
Nov 03 09:45:04.013187 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:04.013168    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894985-wv4z2" event={"ID":"a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5","Type":"ContainerStarted","Data":"2480dee2a2a192b1538009845832748f2b750bf741372ad3fe1c6a6c4726d095"}
Nov 03 09:45:04.014852 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:04.014835    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894985-jp75m" event={"ID":"be2cbd18-00fc-42ff-84d8-1bed98edeb60","Type":"ContainerStarted","Data":"e6aa7dbee05077af8347d08b3c85aecdf9fcdf91bbfc02e6da3cab9a30eec727"}
Nov 03 09:45:04.020376 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:04.020349    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894985-7kqh9" event={"ID":"4ffb2aa7-f878-47e7-995e-f5db6f14b5a3","Type":"ContainerStarted","Data":"55e5fca4055604471ad79998d676c4925260924a801b8011d637f70d208262cf"}
Nov 03 09:45:04.026932 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:04.026599    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894985-2j8r6" event={"ID":"e8a01825-bdfa-4a32-889f-f1bc9a93cb79","Type":"ContainerStarted","Data":"73cbc80fe769c1748aeffd12dfdcf6e9a644a7d8f75427e89164d4e3f5660e7e"}
Nov 03 09:45:04.029644 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:04.029619    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894985-fblt9" event={"ID":"9ad94c34-df19-4f05-a05d-93b64abe1828","Type":"ContainerStarted","Data":"60ac653f8a35d117dcf7b10716e84155a22e15f49f5aa9f99642723048af92a2"}
Nov 03 09:45:04.036034 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:04.035992    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894985-9pscs" event={"ID":"1cd6af3f-fac0-471b-b368-5da14c939450","Type":"ContainerStarted","Data":"19290e20b0ac59a5f888995d3b9c8c7ca209a2b676fc06edfe844449f6e15716"}
Nov 03 09:45:04.038652 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:04.038039    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894985-cftcl" event={"ID":"b2433454-f4b1-4109-a28c-5106db36e5ee","Type":"ContainerStarted","Data":"6e25649c4208d0fab8d36e9ee05e87047220202a2901cfc380ad8f725101d165"}
Nov 03 09:45:04.040034 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:04.040011    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894985-brvhf" event={"ID":"7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed","Type":"ContainerStarted","Data":"292726b08f57e57debe8568bc4df860cf16d3d0b3987a7f6017eb6818082be55"}
Nov 03 09:45:04.539676 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:04.539640    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872281-29894985-xvvlh"]
Nov 03 09:45:04.544888 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:04.544856    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872281-29894985-xvvlh"]
Nov 03 09:45:04.886824 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:04.886770    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894985-fblt9" podStartSLOduration=4.88675761 podStartE2EDuration="4.88675761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:45:04.885183318 +0000 UTC m=+3929.154246766" watchObservedRunningTime="2026-11-03 09:45:04.88675761 +0000 UTC m=+3929.155821043"
Nov 03 09:45:05.050020 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.049989    4473 generic.go:356] "Generic (PLEG): container finished" podID="be2cbd18-00fc-42ff-84d8-1bed98edeb60" containerID="e6aa7dbee05077af8347d08b3c85aecdf9fcdf91bbfc02e6da3cab9a30eec727" exitCode=60
Nov 03 09:45:05.050234 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.050073    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894985-jp75m" event={"ID":"be2cbd18-00fc-42ff-84d8-1bed98edeb60","Type":"ContainerDied","Data":"e6aa7dbee05077af8347d08b3c85aecdf9fcdf91bbfc02e6da3cab9a30eec727"}
Nov 03 09:45:05.050234 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.050178    4473 scope.go:122] "RemoveContainer" containerID="8f3f5bcfe3231a73adf232382c3ec02ec8a6c950516f98cf16b29a998e5f2e59"
Nov 03 09:45:05.050506 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.050485    4473 scope.go:122] "RemoveContainer" containerID="e6aa7dbee05077af8347d08b3c85aecdf9fcdf91bbfc02e6da3cab9a30eec727"
Nov 03 09:45:05.050860 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:05.050834    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894985-jp75m_odh-ai-gateway-infra(be2cbd18-00fc-42ff-84d8-1bed98edeb60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894985-jp75m" podUID="be2cbd18-00fc-42ff-84d8-1bed98edeb60"
Nov 03 09:45:05.053422 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.053387    4473 generic.go:356] "Generic (PLEG): container finished" podID="4ffb2aa7-f878-47e7-995e-f5db6f14b5a3" containerID="55e5fca4055604471ad79998d676c4925260924a801b8011d637f70d208262cf" exitCode=60
Nov 03 09:45:05.053539 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.053471    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894985-7kqh9" event={"ID":"4ffb2aa7-f878-47e7-995e-f5db6f14b5a3","Type":"ContainerDied","Data":"55e5fca4055604471ad79998d676c4925260924a801b8011d637f70d208262cf"}
Nov 03 09:45:05.053742 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.053726    4473 scope.go:122] "RemoveContainer" containerID="55e5fca4055604471ad79998d676c4925260924a801b8011d637f70d208262cf"
Nov 03 09:45:05.054068 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:05.054044    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894985-7kqh9_odh-ai-gateway-infra(4ffb2aa7-f878-47e7-995e-f5db6f14b5a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894985-7kqh9" podUID="4ffb2aa7-f878-47e7-995e-f5db6f14b5a3"
Nov 03 09:45:05.055927 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.055903    4473 generic.go:356] "Generic (PLEG): container finished" podID="e8a01825-bdfa-4a32-889f-f1bc9a93cb79" containerID="73cbc80fe769c1748aeffd12dfdcf6e9a644a7d8f75427e89164d4e3f5660e7e" exitCode=60
Nov 03 09:45:05.056034 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.055978    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894985-2j8r6" event={"ID":"e8a01825-bdfa-4a32-889f-f1bc9a93cb79","Type":"ContainerDied","Data":"73cbc80fe769c1748aeffd12dfdcf6e9a644a7d8f75427e89164d4e3f5660e7e"}
Nov 03 09:45:05.056271 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.056255    4473 scope.go:122] "RemoveContainer" containerID="73cbc80fe769c1748aeffd12dfdcf6e9a644a7d8f75427e89164d4e3f5660e7e"
Nov 03 09:45:05.056543 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:05.056518    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-77207e-29894985-2j8r6_odh-ai-gateway-infra(e8a01825-bdfa-4a32-889f-f1bc9a93cb79)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894985-2j8r6" podUID="e8a01825-bdfa-4a32-889f-f1bc9a93cb79"
Nov 03 09:45:05.058352 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.058334    4473 generic.go:356] "Generic (PLEG): container finished" podID="9ad94c34-df19-4f05-a05d-93b64abe1828" containerID="60ac653f8a35d117dcf7b10716e84155a22e15f49f5aa9f99642723048af92a2" exitCode=60
Nov 03 09:45:05.058466 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.058432    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894985-fblt9" event={"ID":"9ad94c34-df19-4f05-a05d-93b64abe1828","Type":"ContainerDied","Data":"60ac653f8a35d117dcf7b10716e84155a22e15f49f5aa9f99642723048af92a2"}
Nov 03 09:45:05.058677 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.058659    4473 scope.go:122] "RemoveContainer" containerID="60ac653f8a35d117dcf7b10716e84155a22e15f49f5aa9f99642723048af92a2"
Nov 03 09:45:05.058979 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:05.058925    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-910d50-29894985-fblt9_odh-ai-gateway-infra(9ad94c34-df19-4f05-a05d-93b64abe1828)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894985-fblt9" podUID="9ad94c34-df19-4f05-a05d-93b64abe1828"
Nov 03 09:45:05.062112 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.061582    4473 generic.go:356] "Generic (PLEG): container finished" podID="1cd6af3f-fac0-471b-b368-5da14c939450" containerID="19290e20b0ac59a5f888995d3b9c8c7ca209a2b676fc06edfe844449f6e15716" exitCode=60
Nov 03 09:45:05.062112 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.061720    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894985-9pscs" event={"ID":"1cd6af3f-fac0-471b-b368-5da14c939450","Type":"ContainerDied","Data":"19290e20b0ac59a5f888995d3b9c8c7ca209a2b676fc06edfe844449f6e15716"}
Nov 03 09:45:05.062112 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.061920    4473 scope.go:122] "RemoveContainer" containerID="19290e20b0ac59a5f888995d3b9c8c7ca209a2b676fc06edfe844449f6e15716"
Nov 03 09:45:05.062266 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:05.062216    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894985-9pscs_odh-ai-gateway-infra(1cd6af3f-fac0-471b-b368-5da14c939450)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894985-9pscs" podUID="1cd6af3f-fac0-471b-b368-5da14c939450"
Nov 03 09:45:05.064167 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.064144    4473 generic.go:356] "Generic (PLEG): container finished" podID="b2433454-f4b1-4109-a28c-5106db36e5ee" containerID="6e25649c4208d0fab8d36e9ee05e87047220202a2901cfc380ad8f725101d165" exitCode=60
Nov 03 09:45:05.064282 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.064235    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894985-cftcl" event={"ID":"b2433454-f4b1-4109-a28c-5106db36e5ee","Type":"ContainerDied","Data":"6e25649c4208d0fab8d36e9ee05e87047220202a2901cfc380ad8f725101d165"}
Nov 03 09:45:05.064458 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.064444    4473 scope.go:122] "RemoveContainer" containerID="6e25649c4208d0fab8d36e9ee05e87047220202a2901cfc380ad8f725101d165"
Nov 03 09:45:05.064694 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:05.064664    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894985-cftcl_odh-ai-gateway-infra(b2433454-f4b1-4109-a28c-5106db36e5ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894985-cftcl" podUID="b2433454-f4b1-4109-a28c-5106db36e5ee"
Nov 03 09:45:05.066485 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.066463    4473 generic.go:356] "Generic (PLEG): container finished" podID="7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed" containerID="292726b08f57e57debe8568bc4df860cf16d3d0b3987a7f6017eb6818082be55" exitCode=60
Nov 03 09:45:05.066577 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.066519    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894985-brvhf" event={"ID":"7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed","Type":"ContainerDied","Data":"292726b08f57e57debe8568bc4df860cf16d3d0b3987a7f6017eb6818082be55"}
Nov 03 09:45:05.066849 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.066832    4473 scope.go:122] "RemoveContainer" containerID="292726b08f57e57debe8568bc4df860cf16d3d0b3987a7f6017eb6818082be55"
Nov 03 09:45:05.066973 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.066937    4473 scope.go:122] "RemoveContainer" containerID="e262adaeb6a6ec0b65c56b74bd8740ac780b7e9103707c0907bcda2d24c359dd"
Nov 03 09:45:05.067110 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:05.067087    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894985-brvhf_odh-ai-gateway-infra(7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894985-brvhf" podUID="7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed"
Nov 03 09:45:05.068471 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.068447    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75cc44f8d5-hqht4" event={"ID":"35553ad3-951a-4f65-ba00-559a9175bf40","Type":"ContainerStarted","Data":"f53c63905cb1908b303901a2463992620fd13cda0eb30b4127cde435d14c86d8"}
Nov 03 09:45:05.068599 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.068541    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-75cc44f8d5-hqht4"
Nov 03 09:45:05.070586 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.070566    4473 generic.go:356] "Generic (PLEG): container finished" podID="ea1afde1-03bb-4db5-875f-43845656bf63" containerID="b6be289ac64a255263ef4f45b1dcb303e9179e1ac0fad09eaf54f24d977bf4e7" exitCode=60
Nov 03 09:45:05.070692 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.070630    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af514d-29894985-gzb4m" event={"ID":"ea1afde1-03bb-4db5-875f-43845656bf63","Type":"ContainerDied","Data":"b6be289ac64a255263ef4f45b1dcb303e9179e1ac0fad09eaf54f24d977bf4e7"}
Nov 03 09:45:05.070902 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.070880    4473 scope.go:122] "RemoveContainer" containerID="b6be289ac64a255263ef4f45b1dcb303e9179e1ac0fad09eaf54f24d977bf4e7"
Nov 03 09:45:05.071147 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:05.071126    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-af514d-29894985-gzb4m_odh-ai-gateway-infra(ea1afde1-03bb-4db5-875f-43845656bf63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af514d-29894985-gzb4m" podUID="ea1afde1-03bb-4db5-875f-43845656bf63"
Nov 03 09:45:05.072913 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.072890    4473 generic.go:356] "Generic (PLEG): container finished" podID="7dfd9775-ced2-450a-8cc9-f4de56fafa52" containerID="fe53f101ff034f0e489aed9d9e0bb494adb722d793d7e9a2fb63f129eec83b58" exitCode=60
Nov 03 09:45:05.073057 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.073022    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894985-flfp2" event={"ID":"7dfd9775-ced2-450a-8cc9-f4de56fafa52","Type":"ContainerDied","Data":"fe53f101ff034f0e489aed9d9e0bb494adb722d793d7e9a2fb63f129eec83b58"}
Nov 03 09:45:05.073212 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.073193    4473 scope.go:122] "RemoveContainer" containerID="fe53f101ff034f0e489aed9d9e0bb494adb722d793d7e9a2fb63f129eec83b58"
Nov 03 09:45:05.073466 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:05.073443    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-18f9da-29894985-flfp2_odh-ai-gateway-infra(7dfd9775-ced2-450a-8cc9-f4de56fafa52)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894985-flfp2" podUID="7dfd9775-ced2-450a-8cc9-f4de56fafa52"
Nov 03 09:45:05.075630 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.075607    4473 generic.go:356] "Generic (PLEG): container finished" podID="a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5" containerID="2480dee2a2a192b1538009845832748f2b750bf741372ad3fe1c6a6c4726d095" exitCode=60
Nov 03 09:45:05.075725 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.075642    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894985-wv4z2" event={"ID":"a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5","Type":"ContainerDied","Data":"2480dee2a2a192b1538009845832748f2b750bf741372ad3fe1c6a6c4726d095"}
Nov 03 09:45:05.075978 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.075934    4473 scope.go:122] "RemoveContainer" containerID="2480dee2a2a192b1538009845832748f2b750bf741372ad3fe1c6a6c4726d095"
Nov 03 09:45:05.076145 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:05.076127    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-465707-29894985-wv4z2_odh-ai-gateway-infra(a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894985-wv4z2" podUID="a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5"
Nov 03 09:45:05.080655 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.080635    4473 scope.go:122] "RemoveContainer" containerID="532f4df5a8ff1c486878ee9c29e63b79019e7f0f37fc935842c0f111acd1874d"
Nov 03 09:45:05.094605 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.094586    4473 scope.go:122] "RemoveContainer" containerID="bbcc6bef4c8dcccc9affb6e285d8f71b3e82e107dbc2843cf8603d463a13739f"
Nov 03 09:45:05.110117 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.110093    4473 scope.go:122] "RemoveContainer" containerID="864ef5a7ef9b707cd1b269b9f5a96a3de1e3629daaaac5ba81dc8eeb4d04e41a"
Nov 03 09:45:05.125568 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.125545    4473 scope.go:122] "RemoveContainer" containerID="fa8d0a21d7f245d17e9d86bbea135175d0bb18dbe2013319c9e49c807e415bef"
Nov 03 09:45:05.138809 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.138789    4473 scope.go:122] "RemoveContainer" containerID="d361220f359bf8afa1fb6b48b37648f3e07fde640680c871d06eb3c2775ce84b"
Nov 03 09:45:05.153790 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.153763    4473 scope.go:122] "RemoveContainer" containerID="2ba25c2e9a30f41dab51025e9526eee45139e8a64deca8b2546650f658e10281"
Nov 03 09:45:05.166876 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.166854    4473 scope.go:122] "RemoveContainer" containerID="5769528b43b2229e5e325f9649eebbcfda03d64acc39e6aa66fab68696cc602f"
Nov 03 09:45:05.181298 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:05.179764    4473 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6cd2420cdeaf: secret "agc-cred-6cd2420cdeaf" not found
Nov 03 09:45:05.181298 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:05.179796    4473 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b2395a48f4ab: secret "agc-cred-b2395a48f4ab" not found
Nov 03 09:45:05.181298 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:05.179816    4473 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-75cc44f8d5-hqht4: [secret "agc-cred-6cd2420cdeaf" not found, secret "agc-cred-b2395a48f4ab" not found]
Nov 03 09:45:05.181298 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:05.179881    4473 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35553ad3-951a-4f65-ba00-559a9175bf40-provider-credentials podName:35553ad3-951a-4f65-ba00-559a9175bf40 nodeName:}" failed. No retries permitted until 2026-11-03 09:45:05.67986037 +0000 UTC m=+3929.948923784 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/35553ad3-951a-4f65-ba00-559a9175bf40-provider-credentials") pod "payload-processing-external-model-75cc44f8d5-hqht4" (UID: "35553ad3-951a-4f65-ba00-559a9175bf40") : [secret "agc-cred-6cd2420cdeaf" not found, secret "agc-cred-b2395a48f4ab" not found]
Nov 03 09:45:05.181298 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.181216    4473 scope.go:122] "RemoveContainer" containerID="f00a124843dd34c58d5818dc5627c52fd5cb7ed749cabf9456a756e7b6cc6347"
Nov 03 09:45:05.688272 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:05.688242    4473 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6cd2420cdeaf: secret "agc-cred-6cd2420cdeaf" not found
Nov 03 09:45:05.688272 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:05.688272    4473 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b2395a48f4ab: secret "agc-cred-b2395a48f4ab" not found
Nov 03 09:45:05.688820 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:05.688285    4473 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-75cc44f8d5-hqht4: [secret "agc-cred-6cd2420cdeaf" not found, secret "agc-cred-b2395a48f4ab" not found]
Nov 03 09:45:05.688820 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:05.688352    4473 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35553ad3-951a-4f65-ba00-559a9175bf40-provider-credentials podName:35553ad3-951a-4f65-ba00-559a9175bf40 nodeName:}" failed. No retries permitted until 2026-11-03 09:45:06.688331593 +0000 UTC m=+3930.957395022 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/35553ad3-951a-4f65-ba00-559a9175bf40-provider-credentials") pod "payload-processing-external-model-75cc44f8d5-hqht4" (UID: "35553ad3-951a-4f65-ba00-559a9175bf40") : [secret "agc-cred-6cd2420cdeaf" not found, secret "agc-cred-b2395a48f4ab" not found]
Nov 03 09:45:05.706131 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.706097    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-849fb7c5bb-bzq6t"]
Nov 03 09:45:05.709334 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.709313    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-bzq6t"
Nov 03 09:45:05.742447 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.742418    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6dcf4858f8-nl8xt"]
Nov 03 09:45:05.748356 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.748329    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6dcf4858f8-nl8xt"
Nov 03 09:45:05.811004 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.810977    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6dcf4858f8-nl8xt"]
Nov 03 09:45:05.871235 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.871208    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2423bada-0530-46d9-b873-62f7bd6150c9" path="/var/lib/kubelet/pods/2423bada-0530-46d9-b873-62f7bd6150c9/volumes"
Nov 03 09:45:05.891223 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.891194    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f781be3-24e8-4d07-9199-7e5e9722e43f-plugins-config-volume\") pod \"payload-processing-external-model-6dcf4858f8-nl8xt\" (UID: \"5f781be3-24e8-4d07-9199-7e5e9722e43f\") " pod="models-as-a-service/payload-processing-external-model-6dcf4858f8-nl8xt"
Nov 03 09:45:05.891365 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.891299    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f781be3-24e8-4d07-9199-7e5e9722e43f-provider-credentials\") pod \"payload-processing-external-model-6dcf4858f8-nl8xt\" (UID: \"5f781be3-24e8-4d07-9199-7e5e9722e43f\") " pod="models-as-a-service/payload-processing-external-model-6dcf4858f8-nl8xt"
Nov 03 09:45:05.891365 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.891332    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f781be3-24e8-4d07-9199-7e5e9722e43f-routing-overlay-volume\") pod \"payload-processing-external-model-6dcf4858f8-nl8xt\" (UID: \"5f781be3-24e8-4d07-9199-7e5e9722e43f\") " pod="models-as-a-service/payload-processing-external-model-6dcf4858f8-nl8xt"
Nov 03 09:45:05.891474 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.891449    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg6l4\" (UniqueName: \"kubernetes.io/projected/5f781be3-24e8-4d07-9199-7e5e9722e43f-kube-api-access-wg6l4\") pod \"payload-processing-external-model-6dcf4858f8-nl8xt\" (UID: \"5f781be3-24e8-4d07-9199-7e5e9722e43f\") " pod="models-as-a-service/payload-processing-external-model-6dcf4858f8-nl8xt"
Nov 03 09:45:05.992604 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.992508    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f781be3-24e8-4d07-9199-7e5e9722e43f-provider-credentials\") pod \"payload-processing-external-model-6dcf4858f8-nl8xt\" (UID: \"5f781be3-24e8-4d07-9199-7e5e9722e43f\") " pod="models-as-a-service/payload-processing-external-model-6dcf4858f8-nl8xt"
Nov 03 09:45:05.992772 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:05.992655    4473 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b2395a48f4ab: secret "agc-cred-b2395a48f4ab" not found
Nov 03 09:45:05.992772 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:05.992671    4473 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6dcf4858f8-nl8xt: secret "agc-cred-b2395a48f4ab" not found
Nov 03 09:45:05.992772 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:05.992734    4473 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f781be3-24e8-4d07-9199-7e5e9722e43f-provider-credentials podName:5f781be3-24e8-4d07-9199-7e5e9722e43f nodeName:}" failed. No retries permitted until 2026-11-03 09:45:06.492716333 +0000 UTC m=+3930.761779749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5f781be3-24e8-4d07-9199-7e5e9722e43f-provider-credentials") pod "payload-processing-external-model-6dcf4858f8-nl8xt" (UID: "5f781be3-24e8-4d07-9199-7e5e9722e43f") : secret "agc-cred-b2395a48f4ab" not found
Nov 03 09:45:05.993007 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.992918    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f781be3-24e8-4d07-9199-7e5e9722e43f-routing-overlay-volume\") pod \"payload-processing-external-model-6dcf4858f8-nl8xt\" (UID: \"5f781be3-24e8-4d07-9199-7e5e9722e43f\") " pod="models-as-a-service/payload-processing-external-model-6dcf4858f8-nl8xt"
Nov 03 09:45:05.993196 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.993165    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg6l4\" (UniqueName: \"kubernetes.io/projected/5f781be3-24e8-4d07-9199-7e5e9722e43f-kube-api-access-wg6l4\") pod \"payload-processing-external-model-6dcf4858f8-nl8xt\" (UID: \"5f781be3-24e8-4d07-9199-7e5e9722e43f\") " pod="models-as-a-service/payload-processing-external-model-6dcf4858f8-nl8xt"
Nov 03 09:45:05.993328 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.993276    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f781be3-24e8-4d07-9199-7e5e9722e43f-plugins-config-volume\") pod \"payload-processing-external-model-6dcf4858f8-nl8xt\" (UID: \"5f781be3-24e8-4d07-9199-7e5e9722e43f\") " pod="models-as-a-service/payload-processing-external-model-6dcf4858f8-nl8xt"
Nov 03 09:45:05.993597 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.993575    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f781be3-24e8-4d07-9199-7e5e9722e43f-routing-overlay-volume\") pod \"payload-processing-external-model-6dcf4858f8-nl8xt\" (UID: \"5f781be3-24e8-4d07-9199-7e5e9722e43f\") " pod="models-as-a-service/payload-processing-external-model-6dcf4858f8-nl8xt"
Nov 03 09:45:05.994185 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:05.994159    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f781be3-24e8-4d07-9199-7e5e9722e43f-plugins-config-volume\") pod \"payload-processing-external-model-6dcf4858f8-nl8xt\" (UID: \"5f781be3-24e8-4d07-9199-7e5e9722e43f\") " pod="models-as-a-service/payload-processing-external-model-6dcf4858f8-nl8xt"
Nov 03 09:45:06.053780 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:06.053750    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg6l4\" (UniqueName: \"kubernetes.io/projected/5f781be3-24e8-4d07-9199-7e5e9722e43f-kube-api-access-wg6l4\") pod \"payload-processing-external-model-6dcf4858f8-nl8xt\" (UID: \"5f781be3-24e8-4d07-9199-7e5e9722e43f\") " pod="models-as-a-service/payload-processing-external-model-6dcf4858f8-nl8xt"
Nov 03 09:45:06.083237 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:06.083207    4473 scope.go:122] "RemoveContainer" containerID="e6aa7dbee05077af8347d08b3c85aecdf9fcdf91bbfc02e6da3cab9a30eec727"
Nov 03 09:45:06.083544 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:06.083517    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894985-jp75m_odh-ai-gateway-infra(be2cbd18-00fc-42ff-84d8-1bed98edeb60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894985-jp75m" podUID="be2cbd18-00fc-42ff-84d8-1bed98edeb60"
Nov 03 09:45:06.084791 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:06.084757    4473 scope.go:122] "RemoveContainer" containerID="55e5fca4055604471ad79998d676c4925260924a801b8011d637f70d208262cf"
Nov 03 09:45:06.085061 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:06.085041    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894985-7kqh9_odh-ai-gateway-infra(4ffb2aa7-f878-47e7-995e-f5db6f14b5a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894985-7kqh9" podUID="4ffb2aa7-f878-47e7-995e-f5db6f14b5a3"
Nov 03 09:45:06.086357 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:06.086337    4473 scope.go:122] "RemoveContainer" containerID="73cbc80fe769c1748aeffd12dfdcf6e9a644a7d8f75427e89164d4e3f5660e7e"
Nov 03 09:45:06.086607 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:06.086583    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-77207e-29894985-2j8r6_odh-ai-gateway-infra(e8a01825-bdfa-4a32-889f-f1bc9a93cb79)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894985-2j8r6" podUID="e8a01825-bdfa-4a32-889f-f1bc9a93cb79"
Nov 03 09:45:06.087922 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:06.087908    4473 scope.go:122] "RemoveContainer" containerID="60ac653f8a35d117dcf7b10716e84155a22e15f49f5aa9f99642723048af92a2"
Nov 03 09:45:06.088179 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:06.088155    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-910d50-29894985-fblt9_odh-ai-gateway-infra(9ad94c34-df19-4f05-a05d-93b64abe1828)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894985-fblt9" podUID="9ad94c34-df19-4f05-a05d-93b64abe1828"
Nov 03 09:45:06.089588 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:06.089571    4473 scope.go:122] "RemoveContainer" containerID="19290e20b0ac59a5f888995d3b9c8c7ca209a2b676fc06edfe844449f6e15716"
Nov 03 09:45:06.089783 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:06.089762    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894985-9pscs_odh-ai-gateway-infra(1cd6af3f-fac0-471b-b368-5da14c939450)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894985-9pscs" podUID="1cd6af3f-fac0-471b-b368-5da14c939450"
Nov 03 09:45:06.091307 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:06.091287    4473 scope.go:122] "RemoveContainer" containerID="6e25649c4208d0fab8d36e9ee05e87047220202a2901cfc380ad8f725101d165"
Nov 03 09:45:06.091530 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:06.091508    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894985-cftcl_odh-ai-gateway-infra(b2433454-f4b1-4109-a28c-5106db36e5ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894985-cftcl" podUID="b2433454-f4b1-4109-a28c-5106db36e5ee"
Nov 03 09:45:06.092884 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:06.092867    4473 scope.go:122] "RemoveContainer" containerID="292726b08f57e57debe8568bc4df860cf16d3d0b3987a7f6017eb6818082be55"
Nov 03 09:45:06.093085 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:06.093068    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894985-brvhf_odh-ai-gateway-infra(7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894985-brvhf" podUID="7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed"
Nov 03 09:45:06.094572 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:06.094555    4473 scope.go:122] "RemoveContainer" containerID="b6be289ac64a255263ef4f45b1dcb303e9179e1ac0fad09eaf54f24d977bf4e7"
Nov 03 09:45:06.094748 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:06.094729    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-af514d-29894985-gzb4m_odh-ai-gateway-infra(ea1afde1-03bb-4db5-875f-43845656bf63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af514d-29894985-gzb4m" podUID="ea1afde1-03bb-4db5-875f-43845656bf63"
Nov 03 09:45:06.097515 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:06.097490    4473 scope.go:122] "RemoveContainer" containerID="fe53f101ff034f0e489aed9d9e0bb494adb722d793d7e9a2fb63f129eec83b58"
Nov 03 09:45:06.098018 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:06.097988    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-18f9da-29894985-flfp2_odh-ai-gateway-infra(7dfd9775-ced2-450a-8cc9-f4de56fafa52)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894985-flfp2" podUID="7dfd9775-ced2-450a-8cc9-f4de56fafa52"
Nov 03 09:45:06.109560 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:06.109540    4473 scope.go:122] "RemoveContainer" containerID="2480dee2a2a192b1538009845832748f2b750bf741372ad3fe1c6a6c4726d095"
Nov 03 09:45:06.109832 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:06.109813    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-465707-29894985-wv4z2_odh-ai-gateway-infra(a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894985-wv4z2" podUID="a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5"
Nov 03 09:45:06.499707 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:06.499673    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f781be3-24e8-4d07-9199-7e5e9722e43f-provider-credentials\") pod \"payload-processing-external-model-6dcf4858f8-nl8xt\" (UID: \"5f781be3-24e8-4d07-9199-7e5e9722e43f\") " pod="models-as-a-service/payload-processing-external-model-6dcf4858f8-nl8xt"
Nov 03 09:45:06.499886 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:06.499825    4473 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b2395a48f4ab: secret "agc-cred-b2395a48f4ab" not found
Nov 03 09:45:06.499886 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:06.499843    4473 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6dcf4858f8-nl8xt: secret "agc-cred-b2395a48f4ab" not found
Nov 03 09:45:06.499999 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:06.499921    4473 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f781be3-24e8-4d07-9199-7e5e9722e43f-provider-credentials podName:5f781be3-24e8-4d07-9199-7e5e9722e43f nodeName:}" failed. No retries permitted until 2026-11-03 09:45:07.49990307 +0000 UTC m=+3931.768966502 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5f781be3-24e8-4d07-9199-7e5e9722e43f-provider-credentials") pod "payload-processing-external-model-6dcf4858f8-nl8xt" (UID: "5f781be3-24e8-4d07-9199-7e5e9722e43f") : secret "agc-cred-b2395a48f4ab" not found
Nov 03 09:45:06.702321 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:06.702290    4473 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6cd2420cdeaf: secret "agc-cred-6cd2420cdeaf" not found
Nov 03 09:45:06.702321 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:06.702328    4473 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b2395a48f4ab: secret "agc-cred-b2395a48f4ab" not found
Nov 03 09:45:06.702766 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:06.702341    4473 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-75cc44f8d5-hqht4: [secret "agc-cred-6cd2420cdeaf" not found, secret "agc-cred-b2395a48f4ab" not found]
Nov 03 09:45:06.702766 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:06.702419    4473 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35553ad3-951a-4f65-ba00-559a9175bf40-provider-credentials podName:35553ad3-951a-4f65-ba00-559a9175bf40 nodeName:}" failed. No retries permitted until 2026-11-03 09:45:08.702398671 +0000 UTC m=+3932.971462100 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/35553ad3-951a-4f65-ba00-559a9175bf40-provider-credentials") pod "payload-processing-external-model-75cc44f8d5-hqht4" (UID: "35553ad3-951a-4f65-ba00-559a9175bf40") : [secret "agc-cred-6cd2420cdeaf" not found, secret "agc-cred-b2395a48f4ab" not found]
Nov 03 09:45:06.995036 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:06.994985    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-75cc44f8d5-hqht4" podStartSLOduration=3.994971091 podStartE2EDuration="3.994971091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:45:06.992253219 +0000 UTC m=+3931.261316739" watchObservedRunningTime="2026-11-03 09:45:06.994971091 +0000 UTC m=+3931.264034525"
Nov 03 09:45:07.217436 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:07.217403    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6dcf4858f8-nl8xt"]
Nov 03 09:45:07.217986 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:07.217956    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="models-as-a-service/payload-processing-external-model-6dcf4858f8-nl8xt" podUID="5f781be3-24e8-4d07-9199-7e5e9722e43f"
Nov 03 09:45:07.221227 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:07.221205    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-75cc44f8d5-hqht4"]
Nov 03 09:45:07.512484 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:07.512450    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f781be3-24e8-4d07-9199-7e5e9722e43f-provider-credentials\") pod \"payload-processing-external-model-6dcf4858f8-nl8xt\" (UID: \"5f781be3-24e8-4d07-9199-7e5e9722e43f\") " pod="models-as-a-service/payload-processing-external-model-6dcf4858f8-nl8xt"
Nov 03 09:45:07.512672 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:07.512598    4473 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b2395a48f4ab: secret "agc-cred-b2395a48f4ab" not found
Nov 03 09:45:07.512672 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:07.512616    4473 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6dcf4858f8-nl8xt: secret "agc-cred-b2395a48f4ab" not found
Nov 03 09:45:07.512783 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:07.512676    4473 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f781be3-24e8-4d07-9199-7e5e9722e43f-provider-credentials podName:5f781be3-24e8-4d07-9199-7e5e9722e43f nodeName:}" failed. No retries permitted until 2026-11-03 09:45:09.512660525 +0000 UTC m=+3933.781723938 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5f781be3-24e8-4d07-9199-7e5e9722e43f-provider-credentials") pod "payload-processing-external-model-6dcf4858f8-nl8xt" (UID: "5f781be3-24e8-4d07-9199-7e5e9722e43f") : secret "agc-cred-b2395a48f4ab" not found
Nov 03 09:45:08.150018 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:08.136935    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6dcf4858f8-nl8xt"
Nov 03 09:45:08.153383 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:08.152964    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6dcf4858f8-nl8xt"
Nov 03 09:45:08.233226 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:08.231677    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f781be3-24e8-4d07-9199-7e5e9722e43f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f781be3-24e8-4d07-9199-7e5e9722e43f-plugins-config-volume\") pod \"5f781be3-24e8-4d07-9199-7e5e9722e43f\" (UID: \"5f781be3-24e8-4d07-9199-7e5e9722e43f\") "
Nov 03 09:45:08.233226 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:08.231772    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f781be3-24e8-4d07-9199-7e5e9722e43f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f781be3-24e8-4d07-9199-7e5e9722e43f-routing-overlay-volume\") pod \"5f781be3-24e8-4d07-9199-7e5e9722e43f\" (UID: \"5f781be3-24e8-4d07-9199-7e5e9722e43f\") "
Nov 03 09:45:08.233226 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:08.231968    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f781be3-24e8-4d07-9199-7e5e9722e43f-kube-api-access-wg6l4\" (UniqueName: \"kubernetes.io/projected/5f781be3-24e8-4d07-9199-7e5e9722e43f-kube-api-access-wg6l4\") pod \"5f781be3-24e8-4d07-9199-7e5e9722e43f\" (UID: \"5f781be3-24e8-4d07-9199-7e5e9722e43f\") "
Nov 03 09:45:08.233586 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:08.233563    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f781be3-24e8-4d07-9199-7e5e9722e43f-routing-overlay-volume" pod "5f781be3-24e8-4d07-9199-7e5e9722e43f" (UID: "5f781be3-24e8-4d07-9199-7e5e9722e43f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:45:08.233719 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:08.233692    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f781be3-24e8-4d07-9199-7e5e9722e43f-plugins-config-volume" pod "5f781be3-24e8-4d07-9199-7e5e9722e43f" (UID: "5f781be3-24e8-4d07-9199-7e5e9722e43f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:45:08.235234 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:08.235201    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f781be3-24e8-4d07-9199-7e5e9722e43f-kube-api-access-wg6l4" pod "5f781be3-24e8-4d07-9199-7e5e9722e43f" (UID: "5f781be3-24e8-4d07-9199-7e5e9722e43f"). InnerVolumeSpecName "kube-api-access-wg6l4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:45:08.333772 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:08.333710    4473 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f781be3-24e8-4d07-9199-7e5e9722e43f-routing-overlay-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:45:08.333772 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:08.333762    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg6l4\" (UniqueName: \"kubernetes.io/projected/5f781be3-24e8-4d07-9199-7e5e9722e43f-kube-api-access-wg6l4\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:45:08.333772 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:08.333772    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f781be3-24e8-4d07-9199-7e5e9722e43f-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:45:08.739072 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:08.739042    4473 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6cd2420cdeaf: secret "agc-cred-6cd2420cdeaf" not found
Nov 03 09:45:08.739072 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:08.739070    4473 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b2395a48f4ab: secret "agc-cred-b2395a48f4ab" not found
Nov 03 09:45:08.739072 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:08.739079    4473 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-75cc44f8d5-hqht4: [secret "agc-cred-6cd2420cdeaf" not found, secret "agc-cred-b2395a48f4ab" not found]
Nov 03 09:45:08.739325 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:08.739133    4473 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/35553ad3-951a-4f65-ba00-559a9175bf40-provider-credentials podName:35553ad3-951a-4f65-ba00-559a9175bf40 nodeName:}" failed. No retries permitted until 2026-11-03 09:45:12.739117467 +0000 UTC m=+3937.008180892 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/35553ad3-951a-4f65-ba00-559a9175bf40-provider-credentials") pod "payload-processing-external-model-75cc44f8d5-hqht4" (UID: "35553ad3-951a-4f65-ba00-559a9175bf40") : [secret "agc-cred-6cd2420cdeaf" not found, secret "agc-cred-b2395a48f4ab" not found]
Nov 03 09:45:09.143419 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:09.143389    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6dcf4858f8-nl8xt"
Nov 03 09:45:09.376819 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:09.376780    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6dcf4858f8-nl8xt"]
Nov 03 09:45:09.382280 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:09.382253    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6dcf4858f8-nl8xt"]
Nov 03 09:45:09.550095 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:09.550020    4473 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f781be3-24e8-4d07-9199-7e5e9722e43f-provider-credentials\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:45:09.879721 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:09.879674    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f781be3-24e8-4d07-9199-7e5e9722e43f" path="/var/lib/kubelet/pods/5f781be3-24e8-4d07-9199-7e5e9722e43f/volumes"
Nov 03 09:45:10.707089 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:10.707025    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-bzq6t" podUID="f5dbe18e-50c8-402f-9f14-96e484f8fe67" containerName="payload-processing" containerID="cri-o://762e391f563bf8497230893492d866a2389b0f71a3e72050968f56973c7ed8ad" gracePeriod=25
Nov 03 09:45:10.972166 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:10.972103    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-bzq6t"
Nov 03 09:45:11.070531 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:11.070501    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5dbe18e-50c8-402f-9f14-96e484f8fe67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5dbe18e-50c8-402f-9f14-96e484f8fe67-plugins-config-volume\") pod \"f5dbe18e-50c8-402f-9f14-96e484f8fe67\" (UID: \"f5dbe18e-50c8-402f-9f14-96e484f8fe67\") "
Nov 03 09:45:11.070712 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:11.070568    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5dbe18e-50c8-402f-9f14-96e484f8fe67-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5dbe18e-50c8-402f-9f14-96e484f8fe67-routing-overlay-volume\") pod \"f5dbe18e-50c8-402f-9f14-96e484f8fe67\" (UID: \"f5dbe18e-50c8-402f-9f14-96e484f8fe67\") "
Nov 03 09:45:11.070712 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:11.070669    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5dbe18e-50c8-402f-9f14-96e484f8fe67-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5dbe18e-50c8-402f-9f14-96e484f8fe67-provider-credentials\") pod \"f5dbe18e-50c8-402f-9f14-96e484f8fe67\" (UID: \"f5dbe18e-50c8-402f-9f14-96e484f8fe67\") "
Nov 03 09:45:11.070712 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:11.070693    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5dbe18e-50c8-402f-9f14-96e484f8fe67-kube-api-access-2q5np\" (UniqueName: \"kubernetes.io/projected/f5dbe18e-50c8-402f-9f14-96e484f8fe67-kube-api-access-2q5np\") pod \"f5dbe18e-50c8-402f-9f14-96e484f8fe67\" (UID: \"f5dbe18e-50c8-402f-9f14-96e484f8fe67\") "
Nov 03 09:45:11.071004 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:11.070980    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5dbe18e-50c8-402f-9f14-96e484f8fe67-routing-overlay-volume" pod "f5dbe18e-50c8-402f-9f14-96e484f8fe67" (UID: "f5dbe18e-50c8-402f-9f14-96e484f8fe67"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:45:11.071004 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:11.070992    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5dbe18e-50c8-402f-9f14-96e484f8fe67-plugins-config-volume" pod "f5dbe18e-50c8-402f-9f14-96e484f8fe67" (UID: "f5dbe18e-50c8-402f-9f14-96e484f8fe67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:45:11.071196 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:11.071173    4473 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5dbe18e-50c8-402f-9f14-96e484f8fe67-routing-overlay-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:45:11.071196 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:11.071197    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5dbe18e-50c8-402f-9f14-96e484f8fe67-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:45:11.073165 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:11.073140    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5dbe18e-50c8-402f-9f14-96e484f8fe67-kube-api-access-2q5np" pod "f5dbe18e-50c8-402f-9f14-96e484f8fe67" (UID: "f5dbe18e-50c8-402f-9f14-96e484f8fe67"). InnerVolumeSpecName "kube-api-access-2q5np". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:45:11.073246 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:11.073167    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5dbe18e-50c8-402f-9f14-96e484f8fe67-provider-credentials" pod "f5dbe18e-50c8-402f-9f14-96e484f8fe67" (UID: "f5dbe18e-50c8-402f-9f14-96e484f8fe67"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:45:11.163003 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:11.160277    4473 generic.go:356] "Generic (PLEG): container finished" podID="f5dbe18e-50c8-402f-9f14-96e484f8fe67" containerID="762e391f563bf8497230893492d866a2389b0f71a3e72050968f56973c7ed8ad" exitCode=0
Nov 03 09:45:11.163003 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:11.160391    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-bzq6t" event={"ID":"f5dbe18e-50c8-402f-9f14-96e484f8fe67","Type":"ContainerDied","Data":"762e391f563bf8497230893492d866a2389b0f71a3e72050968f56973c7ed8ad"}
Nov 03 09:45:11.163003 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:11.160473    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-bzq6t" event={"ID":"f5dbe18e-50c8-402f-9f14-96e484f8fe67","Type":"ContainerDied","Data":"5cbe4db1a1783eeb0dbcd0605359d02dc879f8798b48efb042de92c594011a4e"}
Nov 03 09:45:11.163003 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:11.160500    4473 scope.go:122] "RemoveContainer" containerID="762e391f563bf8497230893492d866a2389b0f71a3e72050968f56973c7ed8ad"
Nov 03 09:45:11.163003 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:11.160646    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-849fb7c5bb-bzq6t"
Nov 03 09:45:11.173050 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:11.173021    4473 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5dbe18e-50c8-402f-9f14-96e484f8fe67-provider-credentials\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:45:11.173050 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:11.173054    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q5np\" (UniqueName: \"kubernetes.io/projected/f5dbe18e-50c8-402f-9f14-96e484f8fe67-kube-api-access-2q5np\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:45:11.176043 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:11.176025    4473 scope.go:122] "RemoveContainer" containerID="762e391f563bf8497230893492d866a2389b0f71a3e72050968f56973c7ed8ad"
Nov 03 09:45:11.176324 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:11.176305    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"762e391f563bf8497230893492d866a2389b0f71a3e72050968f56973c7ed8ad\": container with ID starting with 762e391f563bf8497230893492d866a2389b0f71a3e72050968f56973c7ed8ad not found: ID does not exist" containerID="762e391f563bf8497230893492d866a2389b0f71a3e72050968f56973c7ed8ad"
Nov 03 09:45:11.176391 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:11.176332    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"762e391f563bf8497230893492d866a2389b0f71a3e72050968f56973c7ed8ad"} err="failed to get container status \"762e391f563bf8497230893492d866a2389b0f71a3e72050968f56973c7ed8ad\": rpc error: code = NotFound desc = could not find container \"762e391f563bf8497230893492d866a2389b0f71a3e72050968f56973c7ed8ad\": container with ID starting with 762e391f563bf8497230893492d866a2389b0f71a3e72050968f56973c7ed8ad not found: ID does not exist"
Nov 03 09:45:11.330752 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:11.330670    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-849fb7c5bb-bzq6t"]
Nov 03 09:45:11.337816 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:11.337785    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-849fb7c5bb-bzq6t"]
Nov 03 09:45:11.870616 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:11.870579    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5dbe18e-50c8-402f-9f14-96e484f8fe67" path="/var/lib/kubelet/pods/f5dbe18e-50c8-402f-9f14-96e484f8fe67/volumes"
Nov 03 09:45:12.221729 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:12.221633    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-75cc44f8d5-hqht4" podUID="35553ad3-951a-4f65-ba00-559a9175bf40" containerName="payload-processing" containerID="cri-o://f53c63905cb1908b303901a2463992620fd13cda0eb30b4127cde435d14c86d8" gracePeriod=25
Nov 03 09:45:12.493965 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:12.493916    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75cc44f8d5-hqht4"
Nov 03 09:45:12.591700 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:12.591673    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35553ad3-951a-4f65-ba00-559a9175bf40-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35553ad3-951a-4f65-ba00-559a9175bf40-routing-overlay-volume\") pod \"35553ad3-951a-4f65-ba00-559a9175bf40\" (UID: \"35553ad3-951a-4f65-ba00-559a9175bf40\") "
Nov 03 09:45:12.591886 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:12.591757    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35553ad3-951a-4f65-ba00-559a9175bf40-provider-credentials\" (UniqueName: \"kubernetes.io/projected/35553ad3-951a-4f65-ba00-559a9175bf40-provider-credentials\") pod \"35553ad3-951a-4f65-ba00-559a9175bf40\" (UID: \"35553ad3-951a-4f65-ba00-559a9175bf40\") "
Nov 03 09:45:12.591886 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:12.591826    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35553ad3-951a-4f65-ba00-559a9175bf40-kube-api-access-59bdg\" (UniqueName: \"kubernetes.io/projected/35553ad3-951a-4f65-ba00-559a9175bf40-kube-api-access-59bdg\") pod \"35553ad3-951a-4f65-ba00-559a9175bf40\" (UID: \"35553ad3-951a-4f65-ba00-559a9175bf40\") "
Nov 03 09:45:12.591886 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:12.591876    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35553ad3-951a-4f65-ba00-559a9175bf40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35553ad3-951a-4f65-ba00-559a9175bf40-plugins-config-volume\") pod \"35553ad3-951a-4f65-ba00-559a9175bf40\" (UID: \"35553ad3-951a-4f65-ba00-559a9175bf40\") "
Nov 03 09:45:12.592119 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:12.592089    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35553ad3-951a-4f65-ba00-559a9175bf40-routing-overlay-volume" pod "35553ad3-951a-4f65-ba00-559a9175bf40" (UID: "35553ad3-951a-4f65-ba00-559a9175bf40"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:45:12.592336 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:12.592316    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35553ad3-951a-4f65-ba00-559a9175bf40-plugins-config-volume" pod "35553ad3-951a-4f65-ba00-559a9175bf40" (UID: "35553ad3-951a-4f65-ba00-559a9175bf40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:45:12.592535 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:12.592519    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35553ad3-951a-4f65-ba00-559a9175bf40-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:45:12.592577 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:12.592538    4473 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35553ad3-951a-4f65-ba00-559a9175bf40-routing-overlay-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:45:12.594078 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:12.594055    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35553ad3-951a-4f65-ba00-559a9175bf40-provider-credentials" pod "35553ad3-951a-4f65-ba00-559a9175bf40" (UID: "35553ad3-951a-4f65-ba00-559a9175bf40"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:45:12.594195 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:12.594175    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35553ad3-951a-4f65-ba00-559a9175bf40-kube-api-access-59bdg" pod "35553ad3-951a-4f65-ba00-559a9175bf40" (UID: "35553ad3-951a-4f65-ba00-559a9175bf40"). InnerVolumeSpecName "kube-api-access-59bdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:45:12.693442 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:12.693399    4473 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/35553ad3-951a-4f65-ba00-559a9175bf40-provider-credentials\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:45:12.693442 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:12.693432    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59bdg\" (UniqueName: \"kubernetes.io/projected/35553ad3-951a-4f65-ba00-559a9175bf40-kube-api-access-59bdg\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:45:13.177216 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:13.177179    4473 generic.go:356] "Generic (PLEG): container finished" podID="35553ad3-951a-4f65-ba00-559a9175bf40" containerID="f53c63905cb1908b303901a2463992620fd13cda0eb30b4127cde435d14c86d8" exitCode=0
Nov 03 09:45:13.177679 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:13.177225    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75cc44f8d5-hqht4" event={"ID":"35553ad3-951a-4f65-ba00-559a9175bf40","Type":"ContainerDied","Data":"f53c63905cb1908b303901a2463992620fd13cda0eb30b4127cde435d14c86d8"}
Nov 03 09:45:13.177679 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:13.177265    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75cc44f8d5-hqht4" event={"ID":"35553ad3-951a-4f65-ba00-559a9175bf40","Type":"ContainerDied","Data":"7178c37e83d6109e7c36bb1b1a6e73940dfb39903cdae36e32052a67cdd7cf55"}
Nov 03 09:45:13.177679 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:13.177285    4473 scope.go:122] "RemoveContainer" containerID="f53c63905cb1908b303901a2463992620fd13cda0eb30b4127cde435d14c86d8"
Nov 03 09:45:13.177679 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:13.177228    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75cc44f8d5-hqht4"
Nov 03 09:45:13.187619 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:13.187598    4473 scope.go:122] "RemoveContainer" containerID="f53c63905cb1908b303901a2463992620fd13cda0eb30b4127cde435d14c86d8"
Nov 03 09:45:13.187959 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:13.187917    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f53c63905cb1908b303901a2463992620fd13cda0eb30b4127cde435d14c86d8\": container with ID starting with f53c63905cb1908b303901a2463992620fd13cda0eb30b4127cde435d14c86d8 not found: ID does not exist" containerID="f53c63905cb1908b303901a2463992620fd13cda0eb30b4127cde435d14c86d8"
Nov 03 09:45:13.188093 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:13.187972    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f53c63905cb1908b303901a2463992620fd13cda0eb30b4127cde435d14c86d8"} err="failed to get container status \"f53c63905cb1908b303901a2463992620fd13cda0eb30b4127cde435d14c86d8\": rpc error: code = NotFound desc = could not find container \"f53c63905cb1908b303901a2463992620fd13cda0eb30b4127cde435d14c86d8\": container with ID starting with f53c63905cb1908b303901a2463992620fd13cda0eb30b4127cde435d14c86d8 not found: ID does not exist"
Nov 03 09:45:13.346543 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:13.346509    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-75cc44f8d5-hqht4"]
Nov 03 09:45:13.353373 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:13.353349    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-75cc44f8d5-hqht4"]
Nov 03 09:45:13.870818 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:13.870784    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35553ad3-951a-4f65-ba00-559a9175bf40" path="/var/lib/kubelet/pods/35553ad3-951a-4f65-ba00-559a9175bf40/volumes"
Nov 03 09:45:14.863063 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:14.863031    4473 scope.go:122] "RemoveContainer" containerID="292726b08f57e57debe8568bc4df860cf16d3d0b3987a7f6017eb6818082be55"
Nov 03 09:45:15.196157 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:15.196124    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894985-brvhf" event={"ID":"7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed","Type":"ContainerStarted","Data":"293aa739e94ba35b7d68c30544635ca55432cd30ccbbfc702732f0f7b81ee28d"}
Nov 03 09:45:15.866835 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:15.866807    4473 scope.go:122] "RemoveContainer" containerID="73cbc80fe769c1748aeffd12dfdcf6e9a644a7d8f75427e89164d4e3f5660e7e"
Nov 03 09:45:15.867350 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:15.866935    4473 scope.go:122] "RemoveContainer" containerID="60ac653f8a35d117dcf7b10716e84155a22e15f49f5aa9f99642723048af92a2"
Nov 03 09:45:16.204061 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:16.204029    4473 generic.go:356] "Generic (PLEG): container finished" podID="7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed" containerID="293aa739e94ba35b7d68c30544635ca55432cd30ccbbfc702732f0f7b81ee28d" exitCode=60
Nov 03 09:45:16.204223 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:16.204097    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894985-brvhf" event={"ID":"7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed","Type":"ContainerDied","Data":"293aa739e94ba35b7d68c30544635ca55432cd30ccbbfc702732f0f7b81ee28d"}
Nov 03 09:45:16.204223 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:16.204162    4473 scope.go:122] "RemoveContainer" containerID="292726b08f57e57debe8568bc4df860cf16d3d0b3987a7f6017eb6818082be55"
Nov 03 09:45:16.204557 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:16.204540    4473 scope.go:122] "RemoveContainer" containerID="293aa739e94ba35b7d68c30544635ca55432cd30ccbbfc702732f0f7b81ee28d"
Nov 03 09:45:16.204872 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:16.204841    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894985-brvhf_odh-ai-gateway-infra(7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894985-brvhf" podUID="7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed"
Nov 03 09:45:16.373707 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:16.373679    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894985-brvhf"]
Nov 03 09:45:16.863684 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:16.863654    4473 scope.go:122] "RemoveContainer" containerID="e6aa7dbee05077af8347d08b3c85aecdf9fcdf91bbfc02e6da3cab9a30eec727"
Nov 03 09:45:16.863863 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:16.863796    4473 scope.go:122] "RemoveContainer" containerID="fe53f101ff034f0e489aed9d9e0bb494adb722d793d7e9a2fb63f129eec83b58"
Nov 03 09:45:16.863863 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:16.863843    4473 scope.go:122] "RemoveContainer" containerID="b6be289ac64a255263ef4f45b1dcb303e9179e1ac0fad09eaf54f24d977bf4e7"
Nov 03 09:45:16.864086 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:16.864062    4473 scope.go:122] "RemoveContainer" containerID="6e25649c4208d0fab8d36e9ee05e87047220202a2901cfc380ad8f725101d165"
Nov 03 09:45:16.864217 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:16.864202    4473 scope.go:122] "RemoveContainer" containerID="19290e20b0ac59a5f888995d3b9c8c7ca209a2b676fc06edfe844449f6e15716"
Nov 03 09:45:16.938375 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:16.938348    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:45:17.125137 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:17.125110    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-af514d-854f5df6d5-7xbnz"]
Nov 03 09:45:17.217362 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:17.217324    4473 generic.go:356] "Generic (PLEG): container finished" podID="e8a01825-bdfa-4a32-889f-f1bc9a93cb79" containerID="9b7c01abdfcbf326b4bbcd01011f451947152ecf26c0f6686fdcd263d7e2e645" exitCode=60
Nov 03 09:45:17.217502 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:17.217424    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894985-2j8r6" event={"ID":"e8a01825-bdfa-4a32-889f-f1bc9a93cb79","Type":"ContainerDied","Data":"9b7c01abdfcbf326b4bbcd01011f451947152ecf26c0f6686fdcd263d7e2e645"}
Nov 03 09:45:17.217502 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:17.217466    4473 scope.go:122] "RemoveContainer" containerID="73cbc80fe769c1748aeffd12dfdcf6e9a644a7d8f75427e89164d4e3f5660e7e"
Nov 03 09:45:17.217824 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:17.217809    4473 scope.go:122] "RemoveContainer" containerID="9b7c01abdfcbf326b4bbcd01011f451947152ecf26c0f6686fdcd263d7e2e645"
Nov 03 09:45:17.218215 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:17.218189    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-77207e-29894985-2j8r6_odh-ai-gateway-infra(e8a01825-bdfa-4a32-889f-f1bc9a93cb79)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894985-2j8r6" podUID="e8a01825-bdfa-4a32-889f-f1bc9a93cb79"
Nov 03 09:45:17.221224 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:17.221186    4473 generic.go:356] "Generic (PLEG): container finished" podID="9ad94c34-df19-4f05-a05d-93b64abe1828" containerID="448e6288514c6de1cdd6f2b065a0ab5aa59c8b7068c450e6d377704dae863f21" exitCode=60
Nov 03 09:45:17.221360 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:17.221244    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894985-fblt9" event={"ID":"9ad94c34-df19-4f05-a05d-93b64abe1828","Type":"ContainerDied","Data":"448e6288514c6de1cdd6f2b065a0ab5aa59c8b7068c450e6d377704dae863f21"}
Nov 03 09:45:17.221641 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:17.221625    4473 scope.go:122] "RemoveContainer" containerID="448e6288514c6de1cdd6f2b065a0ab5aa59c8b7068c450e6d377704dae863f21"
Nov 03 09:45:17.221918 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:17.221877    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-910d50-29894985-fblt9_odh-ai-gateway-infra(9ad94c34-df19-4f05-a05d-93b64abe1828)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894985-fblt9" podUID="9ad94c34-df19-4f05-a05d-93b64abe1828"
Nov 03 09:45:17.251443 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:17.251423    4473 scope.go:122] "RemoveContainer" containerID="60ac653f8a35d117dcf7b10716e84155a22e15f49f5aa9f99642723048af92a2"
Nov 03 09:45:17.388160 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:17.388096    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894985-brvhf"
Nov 03 09:45:17.463125 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:17.463095    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed-kube-api-access-7xj88\" (UniqueName: \"kubernetes.io/projected/7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed-kube-api-access-7xj88\") pod \"7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed\" (UID: \"7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed\") "
Nov 03 09:45:17.463273 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:17.463167    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed-maas-api-service-ca\") pod \"7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed\" (UID: \"7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed\") "
Nov 03 09:45:17.463510 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:17.463487    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed-maas-api-service-ca" pod "7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed" (UID: "7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:45:17.463762 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:17.463739    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed-maas-api-service-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:45:17.465334 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:17.465316    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed-kube-api-access-7xj88" pod "7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed" (UID: "7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed"). InnerVolumeSpecName "kube-api-access-7xj88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:45:17.565241 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:17.565204    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xj88\" (UniqueName: \"kubernetes.io/projected/7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed-kube-api-access-7xj88\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:45:17.863443 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:17.863378    4473 scope.go:122] "RemoveContainer" containerID="55e5fca4055604471ad79998d676c4925260924a801b8011d637f70d208262cf"
Nov 03 09:45:18.229847 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:18.229817    4473 generic.go:356] "Generic (PLEG): container finished" podID="1cd6af3f-fac0-471b-b368-5da14c939450" containerID="a7bd7e8b1df286d2c0d7a6819d48a70a2d274580609bed64fdaa37deb1886473" exitCode=60
Nov 03 09:45:18.230331 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:18.229884    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894985-9pscs" event={"ID":"1cd6af3f-fac0-471b-b368-5da14c939450","Type":"ContainerDied","Data":"a7bd7e8b1df286d2c0d7a6819d48a70a2d274580609bed64fdaa37deb1886473"}
Nov 03 09:45:18.230331 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:18.229939    4473 scope.go:122] "RemoveContainer" containerID="19290e20b0ac59a5f888995d3b9c8c7ca209a2b676fc06edfe844449f6e15716"
Nov 03 09:45:18.230442 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:18.230402    4473 scope.go:122] "RemoveContainer" containerID="a7bd7e8b1df286d2c0d7a6819d48a70a2d274580609bed64fdaa37deb1886473"
Nov 03 09:45:18.230720 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:18.230697    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894985-9pscs_odh-ai-gateway-infra(1cd6af3f-fac0-471b-b368-5da14c939450)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894985-9pscs" podUID="1cd6af3f-fac0-471b-b368-5da14c939450"
Nov 03 09:45:18.231818 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:18.231788    4473 generic.go:356] "Generic (PLEG): container finished" podID="b2433454-f4b1-4109-a28c-5106db36e5ee" containerID="071c8def6ee8339868dca7e44b27892c0058bbe91abf8f45cb04ceadbe795cf3" exitCode=60
Nov 03 09:45:18.231931 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:18.231859    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894985-cftcl" event={"ID":"b2433454-f4b1-4109-a28c-5106db36e5ee","Type":"ContainerDied","Data":"071c8def6ee8339868dca7e44b27892c0058bbe91abf8f45cb04ceadbe795cf3"}
Nov 03 09:45:18.232212 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:18.232196    4473 scope.go:122] "RemoveContainer" containerID="071c8def6ee8339868dca7e44b27892c0058bbe91abf8f45cb04ceadbe795cf3"
Nov 03 09:45:18.232457 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:18.232434    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894985-cftcl_odh-ai-gateway-infra(b2433454-f4b1-4109-a28c-5106db36e5ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894985-cftcl" podUID="b2433454-f4b1-4109-a28c-5106db36e5ee"
Nov 03 09:45:18.233173 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:18.233159    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894985-brvhf"
Nov 03 09:45:18.233255 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:18.233195    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894985-brvhf" event={"ID":"7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed","Type":"ContainerDied","Data":"42ed0c988557a4bfb433abb19c4fa7c4f4f3edc9781b1f94725d0909d5e39c22"}
Nov 03 09:45:18.234982 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:18.234956    4473 generic.go:356] "Generic (PLEG): container finished" podID="ea1afde1-03bb-4db5-875f-43845656bf63" containerID="ff6d2b47782a8b8d82135edaff65545912e4227961a214937228a7a8d8e8d7cd" exitCode=60
Nov 03 09:45:18.235093 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:18.234990    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af514d-29894985-gzb4m" event={"ID":"ea1afde1-03bb-4db5-875f-43845656bf63","Type":"ContainerDied","Data":"ff6d2b47782a8b8d82135edaff65545912e4227961a214937228a7a8d8e8d7cd"}
Nov 03 09:45:18.235307 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:18.235293    4473 scope.go:122] "RemoveContainer" containerID="ff6d2b47782a8b8d82135edaff65545912e4227961a214937228a7a8d8e8d7cd"
Nov 03 09:45:18.235513 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:18.235495    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-af514d-29894985-gzb4m_odh-ai-gateway-infra(ea1afde1-03bb-4db5-875f-43845656bf63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af514d-29894985-gzb4m" podUID="ea1afde1-03bb-4db5-875f-43845656bf63"
Nov 03 09:45:18.236958 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:18.236922    4473 generic.go:356] "Generic (PLEG): container finished" podID="7dfd9775-ced2-450a-8cc9-f4de56fafa52" containerID="dd575050449395831d9be7731de06674c9b082e987dc85c12cc00c16e46367c2" exitCode=60
Nov 03 09:45:18.237044 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:18.236973    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894985-flfp2" event={"ID":"7dfd9775-ced2-450a-8cc9-f4de56fafa52","Type":"ContainerDied","Data":"dd575050449395831d9be7731de06674c9b082e987dc85c12cc00c16e46367c2"}
Nov 03 09:45:18.237294 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:18.237273    4473 scope.go:122] "RemoveContainer" containerID="dd575050449395831d9be7731de06674c9b082e987dc85c12cc00c16e46367c2"
Nov 03 09:45:18.237520 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:18.237502    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-18f9da-29894985-flfp2_odh-ai-gateway-infra(7dfd9775-ced2-450a-8cc9-f4de56fafa52)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894985-flfp2" podUID="7dfd9775-ced2-450a-8cc9-f4de56fafa52"
Nov 03 09:45:18.238862 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:18.238840    4473 generic.go:356] "Generic (PLEG): container finished" podID="be2cbd18-00fc-42ff-84d8-1bed98edeb60" containerID="3603c3878680c94c34491f9f21fa95757cc141aff7bc5f516de91a34ba3f0139" exitCode=60
Nov 03 09:45:18.238978 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:18.238914    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894985-jp75m" event={"ID":"be2cbd18-00fc-42ff-84d8-1bed98edeb60","Type":"ContainerDied","Data":"3603c3878680c94c34491f9f21fa95757cc141aff7bc5f516de91a34ba3f0139"}
Nov 03 09:45:18.239241 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:18.239225    4473 scope.go:122] "RemoveContainer" containerID="3603c3878680c94c34491f9f21fa95757cc141aff7bc5f516de91a34ba3f0139"
Nov 03 09:45:18.239464 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:18.239440    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894985-jp75m_odh-ai-gateway-infra(be2cbd18-00fc-42ff-84d8-1bed98edeb60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894985-jp75m" podUID="be2cbd18-00fc-42ff-84d8-1bed98edeb60"
Nov 03 09:45:18.247468 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:18.247450    4473 scope.go:122] "RemoveContainer" containerID="6e25649c4208d0fab8d36e9ee05e87047220202a2901cfc380ad8f725101d165"
Nov 03 09:45:18.260535 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:18.260517    4473 scope.go:122] "RemoveContainer" containerID="293aa739e94ba35b7d68c30544635ca55432cd30ccbbfc702732f0f7b81ee28d"
Nov 03 09:45:18.268721 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:18.268706    4473 scope.go:122] "RemoveContainer" containerID="b6be289ac64a255263ef4f45b1dcb303e9179e1ac0fad09eaf54f24d977bf4e7"
Nov 03 09:45:18.283521 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:18.283504    4473 scope.go:122] "RemoveContainer" containerID="fe53f101ff034f0e489aed9d9e0bb494adb722d793d7e9a2fb63f129eec83b58"
Nov 03 09:45:18.294827 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:18.294803    4473 scope.go:122] "RemoveContainer" containerID="e6aa7dbee05077af8347d08b3c85aecdf9fcdf91bbfc02e6da3cab9a30eec727"
Nov 03 09:45:18.488141 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:18.488064    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894985-fblt9"]
Nov 03 09:45:18.580234 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:18.580202    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894985-brvhf"]
Nov 03 09:45:18.589851 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:18.589823    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f9b517-29894985-brvhf"]
Nov 03 09:45:18.604191 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:18.604164    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894985-2j8r6"]
Nov 03 09:45:18.628221 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:18.628199    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894985-fblt9"
Nov 03 09:45:18.681973 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:18.681256    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ad94c34-df19-4f05-a05d-93b64abe1828-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ad94c34-df19-4f05-a05d-93b64abe1828-maas-api-service-ca\") pod \"9ad94c34-df19-4f05-a05d-93b64abe1828\" (UID: \"9ad94c34-df19-4f05-a05d-93b64abe1828\") "
Nov 03 09:45:18.681973 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:18.681673    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ad94c34-df19-4f05-a05d-93b64abe1828-maas-api-service-ca" pod "9ad94c34-df19-4f05-a05d-93b64abe1828" (UID: "9ad94c34-df19-4f05-a05d-93b64abe1828"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:45:18.681973 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:18.681861    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ad94c34-df19-4f05-a05d-93b64abe1828-kube-api-access-h9x66\" (UniqueName: \"kubernetes.io/projected/9ad94c34-df19-4f05-a05d-93b64abe1828-kube-api-access-h9x66\") pod \"9ad94c34-df19-4f05-a05d-93b64abe1828\" (UID: \"9ad94c34-df19-4f05-a05d-93b64abe1828\") "
Nov 03 09:45:18.685973 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:18.683759    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ad94c34-df19-4f05-a05d-93b64abe1828-maas-api-service-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:45:18.698463 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:18.698408    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ad94c34-df19-4f05-a05d-93b64abe1828-kube-api-access-h9x66" pod "9ad94c34-df19-4f05-a05d-93b64abe1828" (UID: "9ad94c34-df19-4f05-a05d-93b64abe1828"). InnerVolumeSpecName "kube-api-access-h9x66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:45:18.784964 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:18.784877    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9x66\" (UniqueName: \"kubernetes.io/projected/9ad94c34-df19-4f05-a05d-93b64abe1828-kube-api-access-h9x66\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:45:18.802723 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:18.802698    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894985-2j8r6"
Nov 03 09:45:18.862787 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:18.862751    4473 scope.go:122] "RemoveContainer" containerID="2480dee2a2a192b1538009845832748f2b750bf741372ad3fe1c6a6c4726d095"
Nov 03 09:45:18.920183 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:18.909536    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8a01825-bdfa-4a32-889f-f1bc9a93cb79-kube-api-access-2hz8n\" (UniqueName: \"kubernetes.io/projected/e8a01825-bdfa-4a32-889f-f1bc9a93cb79-kube-api-access-2hz8n\") pod \"e8a01825-bdfa-4a32-889f-f1bc9a93cb79\" (UID: \"e8a01825-bdfa-4a32-889f-f1bc9a93cb79\") "
Nov 03 09:45:18.920183 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:18.909629    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8a01825-bdfa-4a32-889f-f1bc9a93cb79-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a01825-bdfa-4a32-889f-f1bc9a93cb79-maas-api-service-ca\") pod \"e8a01825-bdfa-4a32-889f-f1bc9a93cb79\" (UID: \"e8a01825-bdfa-4a32-889f-f1bc9a93cb79\") "
Nov 03 09:45:18.920183 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:18.913307    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a01825-bdfa-4a32-889f-f1bc9a93cb79-maas-api-service-ca" pod "e8a01825-bdfa-4a32-889f-f1bc9a93cb79" (UID: "e8a01825-bdfa-4a32-889f-f1bc9a93cb79"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:45:19.016013 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.015927    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8a01825-bdfa-4a32-889f-f1bc9a93cb79-maas-api-service-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:45:19.041007 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.040925    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a01825-bdfa-4a32-889f-f1bc9a93cb79-kube-api-access-2hz8n" pod "e8a01825-bdfa-4a32-889f-f1bc9a93cb79" (UID: "e8a01825-bdfa-4a32-889f-f1bc9a93cb79"). InnerVolumeSpecName "kube-api-access-2hz8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:45:19.117858 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.117822    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hz8n\" (UniqueName: \"kubernetes.io/projected/e8a01825-bdfa-4a32-889f-f1bc9a93cb79-kube-api-access-2hz8n\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:45:19.267293 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.267261    4473 generic.go:356] "Generic (PLEG): container finished" podID="4ffb2aa7-f878-47e7-995e-f5db6f14b5a3" containerID="ec60b708a2dac5bc51b27b2003b3bc2d90b665cf6c2af62f476b51a15caf078f" exitCode=60
Nov 03 09:45:19.267813 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.267324    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894985-7kqh9" event={"ID":"4ffb2aa7-f878-47e7-995e-f5db6f14b5a3","Type":"ContainerDied","Data":"ec60b708a2dac5bc51b27b2003b3bc2d90b665cf6c2af62f476b51a15caf078f"}
Nov 03 09:45:19.267813 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.267382    4473 scope.go:122] "RemoveContainer" containerID="55e5fca4055604471ad79998d676c4925260924a801b8011d637f70d208262cf"
Nov 03 09:45:19.267813 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.267800    4473 scope.go:122] "RemoveContainer" containerID="ec60b708a2dac5bc51b27b2003b3bc2d90b665cf6c2af62f476b51a15caf078f"
Nov 03 09:45:19.268161 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:19.268134    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894985-7kqh9_odh-ai-gateway-infra(4ffb2aa7-f878-47e7-995e-f5db6f14b5a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894985-7kqh9" podUID="4ffb2aa7-f878-47e7-995e-f5db6f14b5a3"
Nov 03 09:45:19.269010 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.268982    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894985-2j8r6" event={"ID":"e8a01825-bdfa-4a32-889f-f1bc9a93cb79","Type":"ContainerDied","Data":"c718e13d736b5218780dfb2328cb1aa5284f5896cb6d16f6de2322eec0ba11f0"}
Nov 03 09:45:19.269010 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.269004    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894985-2j8r6"
Nov 03 09:45:19.270537 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.270518    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894985-fblt9" event={"ID":"9ad94c34-df19-4f05-a05d-93b64abe1828","Type":"ContainerDied","Data":"0a480a2742389c32a28853fa4ad2b0d92e1175d840f5541267b618471a8fa413"}
Nov 03 09:45:19.270659 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.270610    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894985-fblt9"
Nov 03 09:45:19.281154 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.281132    4473 scope.go:122] "RemoveContainer" containerID="9b7c01abdfcbf326b4bbcd01011f451947152ecf26c0f6686fdcd263d7e2e645"
Nov 03 09:45:19.290563 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.290542    4473 scope.go:122] "RemoveContainer" containerID="448e6288514c6de1cdd6f2b065a0ab5aa59c8b7068c450e6d377704dae863f21"
Nov 03 09:45:19.365303 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.365276    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894985-9pscs"]
Nov 03 09:45:19.477311 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.477282    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894985-flfp2"]
Nov 03 09:45:19.505722 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.505698    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894985-fblt9"]
Nov 03 09:45:19.514909 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.514881    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-910d50-29894985-fblt9"]
Nov 03 09:45:19.541010 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.540888    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894985-9pscs"
Nov 03 09:45:19.624431 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.623893    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cd6af3f-fac0-471b-b368-5da14c939450-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd6af3f-fac0-471b-b368-5da14c939450-maas-api-service-ca\") pod \"1cd6af3f-fac0-471b-b368-5da14c939450\" (UID: \"1cd6af3f-fac0-471b-b368-5da14c939450\") "
Nov 03 09:45:19.624431 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.624169    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cd6af3f-fac0-471b-b368-5da14c939450-kube-api-access-b5rcx\" (UniqueName: \"kubernetes.io/projected/1cd6af3f-fac0-471b-b368-5da14c939450-kube-api-access-b5rcx\") pod \"1cd6af3f-fac0-471b-b368-5da14c939450\" (UID: \"1cd6af3f-fac0-471b-b368-5da14c939450\") "
Nov 03 09:45:19.624629 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.624501    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cd6af3f-fac0-471b-b368-5da14c939450-maas-api-service-ca" pod "1cd6af3f-fac0-471b-b368-5da14c939450" (UID: "1cd6af3f-fac0-471b-b368-5da14c939450"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:45:19.624671 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.624640    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd6af3f-fac0-471b-b368-5da14c939450-maas-api-service-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:45:19.626678 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.626648    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cd6af3f-fac0-471b-b368-5da14c939450-kube-api-access-b5rcx" pod "1cd6af3f-fac0-471b-b368-5da14c939450" (UID: "1cd6af3f-fac0-471b-b368-5da14c939450"). InnerVolumeSpecName "kube-api-access-b5rcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:45:19.628819 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.628795    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894985-2j8r6"]
Nov 03 09:45:19.631607 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.631586    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-77207e-29894985-2j8r6"]
Nov 03 09:45:19.631682 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.631588    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894985-flfp2"
Nov 03 09:45:19.721994 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.721959    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894985-cftcl"]
Nov 03 09:45:19.725893 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.725870    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dfd9775-ced2-450a-8cc9-f4de56fafa52-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dfd9775-ced2-450a-8cc9-f4de56fafa52-maas-api-service-ca\") pod \"7dfd9775-ced2-450a-8cc9-f4de56fafa52\" (UID: \"7dfd9775-ced2-450a-8cc9-f4de56fafa52\") "
Nov 03 09:45:19.726015 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.725926    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dfd9775-ced2-450a-8cc9-f4de56fafa52-kube-api-access-694gf\" (UniqueName: \"kubernetes.io/projected/7dfd9775-ced2-450a-8cc9-f4de56fafa52-kube-api-access-694gf\") pod \"7dfd9775-ced2-450a-8cc9-f4de56fafa52\" (UID: \"7dfd9775-ced2-450a-8cc9-f4de56fafa52\") "
Nov 03 09:45:19.726248 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.726228    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dfd9775-ced2-450a-8cc9-f4de56fafa52-maas-api-service-ca" pod "7dfd9775-ced2-450a-8cc9-f4de56fafa52" (UID: "7dfd9775-ced2-450a-8cc9-f4de56fafa52"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:45:19.726492 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.726470    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dfd9775-ced2-450a-8cc9-f4de56fafa52-maas-api-service-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:45:19.726789 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.726498    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5rcx\" (UniqueName: \"kubernetes.io/projected/1cd6af3f-fac0-471b-b368-5da14c939450-kube-api-access-b5rcx\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:45:19.728476 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.728454    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dfd9775-ced2-450a-8cc9-f4de56fafa52-kube-api-access-694gf" pod "7dfd9775-ced2-450a-8cc9-f4de56fafa52" (UID: "7dfd9775-ced2-450a-8cc9-f4de56fafa52"). InnerVolumeSpecName "kube-api-access-694gf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:45:19.828028 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.827925    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-694gf\" (UniqueName: \"kubernetes.io/projected/7dfd9775-ced2-450a-8cc9-f4de56fafa52-kube-api-access-694gf\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:45:19.848850 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.848818    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894985-jp75m"]
Nov 03 09:45:19.851620 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.851596    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894985-cftcl"
Nov 03 09:45:19.902800 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.898177    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed" path="/var/lib/kubelet/pods/7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed/volumes"
Nov 03 09:45:19.902800 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.899552    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ad94c34-df19-4f05-a05d-93b64abe1828" path="/var/lib/kubelet/pods/9ad94c34-df19-4f05-a05d-93b64abe1828/volumes"
Nov 03 09:45:19.902800 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.900532    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8a01825-bdfa-4a32-889f-f1bc9a93cb79" path="/var/lib/kubelet/pods/e8a01825-bdfa-4a32-889f-f1bc9a93cb79/volumes"
Nov 03 09:45:19.933986 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.928749    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2433454-f4b1-4109-a28c-5106db36e5ee-kube-api-access-79r89\" (UniqueName: \"kubernetes.io/projected/b2433454-f4b1-4109-a28c-5106db36e5ee-kube-api-access-79r89\") pod \"b2433454-f4b1-4109-a28c-5106db36e5ee\" (UID: \"b2433454-f4b1-4109-a28c-5106db36e5ee\") "
Nov 03 09:45:19.933986 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.928993    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2433454-f4b1-4109-a28c-5106db36e5ee-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2433454-f4b1-4109-a28c-5106db36e5ee-maas-api-service-ca\") pod \"b2433454-f4b1-4109-a28c-5106db36e5ee\" (UID: \"b2433454-f4b1-4109-a28c-5106db36e5ee\") "
Nov 03 09:45:19.933986 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.929991    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2433454-f4b1-4109-a28c-5106db36e5ee-maas-api-service-ca" pod "b2433454-f4b1-4109-a28c-5106db36e5ee" (UID: "b2433454-f4b1-4109-a28c-5106db36e5ee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:45:19.933986 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.932392    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2433454-f4b1-4109-a28c-5106db36e5ee-kube-api-access-79r89" pod "b2433454-f4b1-4109-a28c-5106db36e5ee" (UID: "b2433454-f4b1-4109-a28c-5106db36e5ee"). InnerVolumeSpecName "kube-api-access-79r89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:45:19.988894 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:19.988863    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af514d-29894985-gzb4m"]
Nov 03 09:45:20.031684 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.031298    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2433454-f4b1-4109-a28c-5106db36e5ee-maas-api-service-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:45:20.031684 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.031332    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79r89\" (UniqueName: \"kubernetes.io/projected/b2433454-f4b1-4109-a28c-5106db36e5ee-kube-api-access-79r89\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:45:20.082636 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.082577    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894985-jp75m"
Nov 03 09:45:20.132839 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.132806    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be2cbd18-00fc-42ff-84d8-1bed98edeb60-kube-api-access-fzv84\" (UniqueName: \"kubernetes.io/projected/be2cbd18-00fc-42ff-84d8-1bed98edeb60-kube-api-access-fzv84\") pod \"be2cbd18-00fc-42ff-84d8-1bed98edeb60\" (UID: \"be2cbd18-00fc-42ff-84d8-1bed98edeb60\") "
Nov 03 09:45:20.133038 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.132880    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be2cbd18-00fc-42ff-84d8-1bed98edeb60-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be2cbd18-00fc-42ff-84d8-1bed98edeb60-maas-api-service-ca\") pod \"be2cbd18-00fc-42ff-84d8-1bed98edeb60\" (UID: \"be2cbd18-00fc-42ff-84d8-1bed98edeb60\") "
Nov 03 09:45:20.133273 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.133248    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be2cbd18-00fc-42ff-84d8-1bed98edeb60-maas-api-service-ca" pod "be2cbd18-00fc-42ff-84d8-1bed98edeb60" (UID: "be2cbd18-00fc-42ff-84d8-1bed98edeb60"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:45:20.133408 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.133370    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be2cbd18-00fc-42ff-84d8-1bed98edeb60-maas-api-service-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:45:20.135237 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.135217    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be2cbd18-00fc-42ff-84d8-1bed98edeb60-kube-api-access-fzv84" pod "be2cbd18-00fc-42ff-84d8-1bed98edeb60" (UID: "be2cbd18-00fc-42ff-84d8-1bed98edeb60"). InnerVolumeSpecName "kube-api-access-fzv84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:45:20.149081 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.149059    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af514d-29894985-gzb4m"
Nov 03 09:45:20.234212 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.234184    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea1afde1-03bb-4db5-875f-43845656bf63-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea1afde1-03bb-4db5-875f-43845656bf63-maas-api-service-ca\") pod \"ea1afde1-03bb-4db5-875f-43845656bf63\" (UID: \"ea1afde1-03bb-4db5-875f-43845656bf63\") "
Nov 03 09:45:20.234407 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.234225    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea1afde1-03bb-4db5-875f-43845656bf63-kube-api-access-zjjs6\" (UniqueName: \"kubernetes.io/projected/ea1afde1-03bb-4db5-875f-43845656bf63-kube-api-access-zjjs6\") pod \"ea1afde1-03bb-4db5-875f-43845656bf63\" (UID: \"ea1afde1-03bb-4db5-875f-43845656bf63\") "
Nov 03 09:45:20.234555 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.234529    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea1afde1-03bb-4db5-875f-43845656bf63-maas-api-service-ca" pod "ea1afde1-03bb-4db5-875f-43845656bf63" (UID: "ea1afde1-03bb-4db5-875f-43845656bf63"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:45:20.234713 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.234698    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea1afde1-03bb-4db5-875f-43845656bf63-maas-api-service-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:45:20.234785 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.234715    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzv84\" (UniqueName: \"kubernetes.io/projected/be2cbd18-00fc-42ff-84d8-1bed98edeb60-kube-api-access-fzv84\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:45:20.236553 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.236528    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea1afde1-03bb-4db5-875f-43845656bf63-kube-api-access-zjjs6" pod "ea1afde1-03bb-4db5-875f-43845656bf63" (UID: "ea1afde1-03bb-4db5-875f-43845656bf63"). InnerVolumeSpecName "kube-api-access-zjjs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:45:20.279498 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.279468    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894985-jp75m"
Nov 03 09:45:20.279999 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.279472    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894985-jp75m" event={"ID":"be2cbd18-00fc-42ff-84d8-1bed98edeb60","Type":"ContainerDied","Data":"9ba058b339451e69dee1363206e81fde8a4992397c1717189cd0d0193cc6bea6"}
Nov 03 09:45:20.279999 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.279607    4473 scope.go:122] "RemoveContainer" containerID="3603c3878680c94c34491f9f21fa95757cc141aff7bc5f516de91a34ba3f0139"
Nov 03 09:45:20.284886 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.284863    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894985-9pscs"
Nov 03 09:45:20.285052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.284931    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894985-9pscs" event={"ID":"1cd6af3f-fac0-471b-b368-5da14c939450","Type":"ContainerDied","Data":"cefaa69c90fdabe10c8e1dde0868d2d896686be80c356b0c939bdc845f5709ab"}
Nov 03 09:45:20.288238 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.288217    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894985-cftcl"
Nov 03 09:45:20.288367 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.288279    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894985-cftcl" event={"ID":"b2433454-f4b1-4109-a28c-5106db36e5ee","Type":"ContainerDied","Data":"cfc6feb96516f9e9a671c888d30d2c03ad76e54384c70f0977f4d8733d564ba8"}
Nov 03 09:45:20.289814 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.289794    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af514d-29894985-gzb4m" event={"ID":"ea1afde1-03bb-4db5-875f-43845656bf63","Type":"ContainerDied","Data":"f39dc76bd456a706240d41a7646606e1e998bf12f2dddd6f73a32758161959e9"}
Nov 03 09:45:20.289928 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.289868    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af514d-29894985-gzb4m"
Nov 03 09:45:20.291010 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.290990    4473 scope.go:122] "RemoveContainer" containerID="a7bd7e8b1df286d2c0d7a6819d48a70a2d274580609bed64fdaa37deb1886473"
Nov 03 09:45:20.291516 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.291494    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894985-flfp2" event={"ID":"7dfd9775-ced2-450a-8cc9-f4de56fafa52","Type":"ContainerDied","Data":"c6d5a25e5079a484268d45456bfa2df3206b50ec3e13e3336049f290a9443443"}
Nov 03 09:45:20.291666 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.291528    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894985-flfp2"
Nov 03 09:45:20.296955 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.296913    4473 generic.go:356] "Generic (PLEG): container finished" podID="a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5" containerID="0246199b10fb926681bda8235f892940852862b6fa3df054150dd8ad53177179" exitCode=60
Nov 03 09:45:20.297074 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.296980    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894985-wv4z2" event={"ID":"a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5","Type":"ContainerDied","Data":"0246199b10fb926681bda8235f892940852862b6fa3df054150dd8ad53177179"}
Nov 03 09:45:20.297409 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.297392    4473 scope.go:122] "RemoveContainer" containerID="0246199b10fb926681bda8235f892940852862b6fa3df054150dd8ad53177179"
Nov 03 09:45:20.297731 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:20.297708    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-465707-29894985-wv4z2_odh-ai-gateway-infra(a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894985-wv4z2" podUID="a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5"
Nov 03 09:45:20.302921 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.302906    4473 scope.go:122] "RemoveContainer" containerID="071c8def6ee8339868dca7e44b27892c0058bbe91abf8f45cb04ceadbe795cf3"
Nov 03 09:45:20.312202 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.312181    4473 scope.go:122] "RemoveContainer" containerID="ff6d2b47782a8b8d82135edaff65545912e4227961a214937228a7a8d8e8d7cd"
Nov 03 09:45:20.320607 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.320589    4473 scope.go:122] "RemoveContainer" containerID="dd575050449395831d9be7731de06674c9b082e987dc85c12cc00c16e46367c2"
Nov 03 09:45:20.332032 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.331497    4473 scope.go:122] "RemoveContainer" containerID="2480dee2a2a192b1538009845832748f2b750bf741372ad3fe1c6a6c4726d095"
Nov 03 09:45:20.336204 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.336181    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjjs6\" (UniqueName: \"kubernetes.io/projected/ea1afde1-03bb-4db5-875f-43845656bf63-kube-api-access-zjjs6\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:45:20.397735 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.397698    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894985-7kqh9"]
Nov 03 09:45:20.533673 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.533649    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894985-7kqh9"
Nov 03 09:45:20.571206 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.571178    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894985-jp75m"]
Nov 03 09:45:20.576419 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.576396    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894985-jp75m"]
Nov 03 09:45:20.639405 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.639379    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ffb2aa7-f878-47e7-995e-f5db6f14b5a3-kube-api-access-4776h\" (UniqueName: \"kubernetes.io/projected/4ffb2aa7-f878-47e7-995e-f5db6f14b5a3-kube-api-access-4776h\") pod \"4ffb2aa7-f878-47e7-995e-f5db6f14b5a3\" (UID: \"4ffb2aa7-f878-47e7-995e-f5db6f14b5a3\") "
Nov 03 09:45:20.639575 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.639428    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ffb2aa7-f878-47e7-995e-f5db6f14b5a3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ffb2aa7-f878-47e7-995e-f5db6f14b5a3-maas-api-service-ca\") pod \"4ffb2aa7-f878-47e7-995e-f5db6f14b5a3\" (UID: \"4ffb2aa7-f878-47e7-995e-f5db6f14b5a3\") "
Nov 03 09:45:20.639793 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.639772    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ffb2aa7-f878-47e7-995e-f5db6f14b5a3-maas-api-service-ca" pod "4ffb2aa7-f878-47e7-995e-f5db6f14b5a3" (UID: "4ffb2aa7-f878-47e7-995e-f5db6f14b5a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:45:20.641560 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.641539    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ffb2aa7-f878-47e7-995e-f5db6f14b5a3-kube-api-access-4776h" pod "4ffb2aa7-f878-47e7-995e-f5db6f14b5a3" (UID: "4ffb2aa7-f878-47e7-995e-f5db6f14b5a3"). InnerVolumeSpecName "kube-api-access-4776h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:45:20.741301 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.741269    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4776h\" (UniqueName: \"kubernetes.io/projected/4ffb2aa7-f878-47e7-995e-f5db6f14b5a3-kube-api-access-4776h\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:45:20.741301 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.741297    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ffb2aa7-f878-47e7-995e-f5db6f14b5a3-maas-api-service-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:45:20.750017 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.749986    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af514d-29894985-gzb4m"]
Nov 03 09:45:20.755967 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.755932    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-af514d-29894985-gzb4m"]
Nov 03 09:45:20.811245 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811212    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f9981fd-t4djq"]
Nov 03 09:45:20.811717 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811704    4473 cpu_manager.go:414] "containerMap: removing container" podUID="f5dbe18e-50c8-402f-9f14-96e484f8fe67" containerName="payload-processing"
Nov 03 09:45:20.811770 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811719    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5dbe18e-50c8-402f-9f14-96e484f8fe67" containerName="payload-processing"
Nov 03 09:45:20.811770 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811733    4473 cpu_manager.go:414] "containerMap: removing container" podUID="e8a01825-bdfa-4a32-889f-f1bc9a93cb79" containerName="cleanup"
Nov 03 09:45:20.811770 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811738    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8a01825-bdfa-4a32-889f-f1bc9a93cb79" containerName="cleanup"
Nov 03 09:45:20.811770 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811744    4473 cpu_manager.go:414] "containerMap: removing container" podUID="ea1afde1-03bb-4db5-875f-43845656bf63" containerName="cleanup"
Nov 03 09:45:20.811770 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811749    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea1afde1-03bb-4db5-875f-43845656bf63" containerName="cleanup"
Nov 03 09:45:20.811770 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811756    4473 cpu_manager.go:414] "containerMap: removing container" podUID="be2cbd18-00fc-42ff-84d8-1bed98edeb60" containerName="cleanup"
Nov 03 09:45:20.811770 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811760    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be2cbd18-00fc-42ff-84d8-1bed98edeb60" containerName="cleanup"
Nov 03 09:45:20.811770 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811768    4473 cpu_manager.go:414] "containerMap: removing container" podUID="1cd6af3f-fac0-471b-b368-5da14c939450" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811773    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cd6af3f-fac0-471b-b368-5da14c939450" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811783    4473 cpu_manager.go:414] "containerMap: removing container" podUID="e8a01825-bdfa-4a32-889f-f1bc9a93cb79" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811788    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8a01825-bdfa-4a32-889f-f1bc9a93cb79" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811795    4473 cpu_manager.go:414] "containerMap: removing container" podUID="e8a01825-bdfa-4a32-889f-f1bc9a93cb79" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811799    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8a01825-bdfa-4a32-889f-f1bc9a93cb79" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811806    4473 cpu_manager.go:414] "containerMap: removing container" podUID="4ffb2aa7-f878-47e7-995e-f5db6f14b5a3" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811811    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ffb2aa7-f878-47e7-995e-f5db6f14b5a3" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811817    4473 cpu_manager.go:414] "containerMap: removing container" podUID="b2433454-f4b1-4109-a28c-5106db36e5ee" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811822    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2433454-f4b1-4109-a28c-5106db36e5ee" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811827    4473 cpu_manager.go:414] "containerMap: removing container" podUID="b2433454-f4b1-4109-a28c-5106db36e5ee" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811832    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2433454-f4b1-4109-a28c-5106db36e5ee" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811843    4473 cpu_manager.go:414] "containerMap: removing container" podUID="7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811848    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811862    4473 cpu_manager.go:414] "containerMap: removing container" podUID="4ffb2aa7-f878-47e7-995e-f5db6f14b5a3" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811866    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ffb2aa7-f878-47e7-995e-f5db6f14b5a3" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811872    4473 cpu_manager.go:414] "containerMap: removing container" podUID="b2433454-f4b1-4109-a28c-5106db36e5ee" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811877    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2433454-f4b1-4109-a28c-5106db36e5ee" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811882    4473 cpu_manager.go:414] "containerMap: removing container" podUID="7dfd9775-ced2-450a-8cc9-f4de56fafa52" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811886    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dfd9775-ced2-450a-8cc9-f4de56fafa52" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811894    4473 cpu_manager.go:414] "containerMap: removing container" podUID="be2cbd18-00fc-42ff-84d8-1bed98edeb60" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811898    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be2cbd18-00fc-42ff-84d8-1bed98edeb60" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811905    4473 cpu_manager.go:414] "containerMap: removing container" podUID="7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811909    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811916    4473 cpu_manager.go:414] "containerMap: removing container" podUID="1cd6af3f-fac0-471b-b368-5da14c939450" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811920    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cd6af3f-fac0-471b-b368-5da14c939450" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811926    4473 cpu_manager.go:414] "containerMap: removing container" podUID="9ad94c34-df19-4f05-a05d-93b64abe1828" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811930    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ad94c34-df19-4f05-a05d-93b64abe1828" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811940    4473 cpu_manager.go:414] "containerMap: removing container" podUID="ea1afde1-03bb-4db5-875f-43845656bf63" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811977    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea1afde1-03bb-4db5-875f-43845656bf63" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.811995    4473 cpu_manager.go:414] "containerMap: removing container" podUID="9ad94c34-df19-4f05-a05d-93b64abe1828" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.812000    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ad94c34-df19-4f05-a05d-93b64abe1828" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.812006    4473 cpu_manager.go:414] "containerMap: removing container" podUID="35553ad3-951a-4f65-ba00-559a9175bf40" containerName="payload-processing"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.812011    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35553ad3-951a-4f65-ba00-559a9175bf40" containerName="payload-processing"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.812021    4473 cpu_manager.go:414] "containerMap: removing container" podUID="7dfd9775-ced2-450a-8cc9-f4de56fafa52" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.812025    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dfd9775-ced2-450a-8cc9-f4de56fafa52" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.812031    4473 cpu_manager.go:414] "containerMap: removing container" podUID="7dfd9775-ced2-450a-8cc9-f4de56fafa52" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.812035    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dfd9775-ced2-450a-8cc9-f4de56fafa52" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.812041    4473 cpu_manager.go:414] "containerMap: removing container" podUID="be2cbd18-00fc-42ff-84d8-1bed98edeb60" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.812045    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be2cbd18-00fc-42ff-84d8-1bed98edeb60" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.812052    4473 cpu_manager.go:414] "containerMap: removing container" podUID="9ad94c34-df19-4f05-a05d-93b64abe1828" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.812056    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ad94c34-df19-4f05-a05d-93b64abe1828" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.812067    4473 cpu_manager.go:414] "containerMap: removing container" podUID="4ffb2aa7-f878-47e7-995e-f5db6f14b5a3" containerName="cleanup"
Nov 03 09:45:20.812052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.812072    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ffb2aa7-f878-47e7-995e-f5db6f14b5a3" containerName="cleanup"
Nov 03 09:45:20.813499 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.812152    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ffb2aa7-f878-47e7-995e-f5db6f14b5a3" containerName="cleanup"
Nov 03 09:45:20.813499 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.812160    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2433454-f4b1-4109-a28c-5106db36e5ee" containerName="cleanup"
Nov 03 09:45:20.813499 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.812166    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dfd9775-ced2-450a-8cc9-f4de56fafa52" containerName="cleanup"
Nov 03 09:45:20.813499 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.812171    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="be2cbd18-00fc-42ff-84d8-1bed98edeb60" containerName="cleanup"
Nov 03 09:45:20.813499 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.812176    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed" containerName="cleanup"
Nov 03 09:45:20.813499 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.812181    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cd6af3f-fac0-471b-b368-5da14c939450" containerName="cleanup"
Nov 03 09:45:20.813499 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.812185    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cd6af3f-fac0-471b-b368-5da14c939450" containerName="cleanup"
Nov 03 09:45:20.813499 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.812191    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ad94c34-df19-4f05-a05d-93b64abe1828" containerName="cleanup"
Nov 03 09:45:20.813499 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.812197    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea1afde1-03bb-4db5-875f-43845656bf63" containerName="cleanup"
Nov 03 09:45:20.813499 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.812202    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ad94c34-df19-4f05-a05d-93b64abe1828" containerName="cleanup"
Nov 03 09:45:20.813499 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.812207    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="35553ad3-951a-4f65-ba00-559a9175bf40" containerName="payload-processing"
Nov 03 09:45:20.813499 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.812211    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dfd9775-ced2-450a-8cc9-f4de56fafa52" containerName="cleanup"
Nov 03 09:45:20.813499 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.812215    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dfd9775-ced2-450a-8cc9-f4de56fafa52" containerName="cleanup"
Nov 03 09:45:20.813499 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.812220    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="be2cbd18-00fc-42ff-84d8-1bed98edeb60" containerName="cleanup"
Nov 03 09:45:20.813499 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.812225    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ad94c34-df19-4f05-a05d-93b64abe1828" containerName="cleanup"
Nov 03 09:45:20.813499 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.812231    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed" containerName="cleanup"
Nov 03 09:45:20.813499 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.812236    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5dbe18e-50c8-402f-9f14-96e484f8fe67" containerName="payload-processing"
Nov 03 09:45:20.813499 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.812241    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8a01825-bdfa-4a32-889f-f1bc9a93cb79" containerName="cleanup"
Nov 03 09:45:20.813499 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.812247    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea1afde1-03bb-4db5-875f-43845656bf63" containerName="cleanup"
Nov 03 09:45:20.813499 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.812252    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="be2cbd18-00fc-42ff-84d8-1bed98edeb60" containerName="cleanup"
Nov 03 09:45:20.813499 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.812258    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8a01825-bdfa-4a32-889f-f1bc9a93cb79" containerName="cleanup"
Nov 03 09:45:20.813499 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.812263    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8a01825-bdfa-4a32-889f-f1bc9a93cb79" containerName="cleanup"
Nov 03 09:45:20.813499 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.812268    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea1afde1-03bb-4db5-875f-43845656bf63" containerName="cleanup"
Nov 03 09:45:20.813499 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.812273    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ffb2aa7-f878-47e7-995e-f5db6f14b5a3" containerName="cleanup"
Nov 03 09:45:20.813499 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.812277    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2433454-f4b1-4109-a28c-5106db36e5ee" containerName="cleanup"
Nov 03 09:45:20.815670 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.815654    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f9981fd-t4djq"
Nov 03 09:45:20.873313 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.873282    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:45:20.873580 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.873553    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f9981fd-t4djq"]
Nov 03 09:45:20.943316 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.943290    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k28g\" (UniqueName: \"kubernetes.io/projected/1b2df33a-75de-45f9-b7c4-45f13ab82990-kube-api-access-9k28g\") pod \"maas-api-revoke-sub-4f9981fd-t4djq\" (UID: \"1b2df33a-75de-45f9-b7c4-45f13ab82990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f9981fd-t4djq"
Nov 03 09:45:20.943455 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.943357    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b2df33a-75de-45f9-b7c4-45f13ab82990-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f9981fd-t4djq\" (UID: \"1b2df33a-75de-45f9-b7c4-45f13ab82990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f9981fd-t4djq"
Nov 03 09:45:20.943504 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.943473    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b2df33a-75de-45f9-b7c4-45f13ab82990-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f9981fd-t4djq\" (UID: \"1b2df33a-75de-45f9-b7c4-45f13ab82990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f9981fd-t4djq"
Nov 03 09:45:20.987305 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.987275    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894985-flfp2"]
Nov 03 09:45:20.992660 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:20.992632    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-18f9da-29894985-flfp2"]
Nov 03 09:45:21.044226 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:21.044195    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b2df33a-75de-45f9-b7c4-45f13ab82990-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f9981fd-t4djq\" (UID: \"1b2df33a-75de-45f9-b7c4-45f13ab82990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f9981fd-t4djq"
Nov 03 09:45:21.044337 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:21.044269    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b2df33a-75de-45f9-b7c4-45f13ab82990-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f9981fd-t4djq\" (UID: \"1b2df33a-75de-45f9-b7c4-45f13ab82990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f9981fd-t4djq"
Nov 03 09:45:21.044400 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:21.044341    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k28g\" (UniqueName: \"kubernetes.io/projected/1b2df33a-75de-45f9-b7c4-45f13ab82990-kube-api-access-9k28g\") pod \"maas-api-revoke-sub-4f9981fd-t4djq\" (UID: \"1b2df33a-75de-45f9-b7c4-45f13ab82990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f9981fd-t4djq"
Nov 03 09:45:21.044878 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:21.044857    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b2df33a-75de-45f9-b7c4-45f13ab82990-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f9981fd-t4djq\" (UID: \"1b2df33a-75de-45f9-b7c4-45f13ab82990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f9981fd-t4djq"
Nov 03 09:45:21.046692 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:21.046670    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b2df33a-75de-45f9-b7c4-45f13ab82990-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f9981fd-t4djq\" (UID: \"1b2df33a-75de-45f9-b7c4-45f13ab82990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f9981fd-t4djq"
Nov 03 09:45:21.104842 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:21.104814    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894985-9pscs"]
Nov 03 09:45:21.107620 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:21.107594    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k28g\" (UniqueName: \"kubernetes.io/projected/1b2df33a-75de-45f9-b7c4-45f13ab82990-kube-api-access-9k28g\") pod \"maas-api-revoke-sub-4f9981fd-t4djq\" (UID: \"1b2df33a-75de-45f9-b7c4-45f13ab82990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f9981fd-t4djq"
Nov 03 09:45:21.107725 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:21.107676    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9ed110-29894985-9pscs"]
Nov 03 09:45:21.128252 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:21.128233    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f9981fd-t4djq"
Nov 03 09:45:21.307755 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:21.307719    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894985-7kqh9" event={"ID":"4ffb2aa7-f878-47e7-995e-f5db6f14b5a3","Type":"ContainerDied","Data":"e9196d40c0e24e34203c625b24027d53c80cae1375db181cd2683cef4e8bda2f"}
Nov 03 09:45:21.307755 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:21.307739    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894985-7kqh9"
Nov 03 09:45:21.308264 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:21.307801    4473 scope.go:122] "RemoveContainer" containerID="ec60b708a2dac5bc51b27b2003b3bc2d90b665cf6c2af62f476b51a15caf078f"
Nov 03 09:45:21.336954 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:21.336919    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894985-cftcl"]
Nov 03 09:45:21.341819 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:21.341799    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-b196c8-29894985-cftcl"]
Nov 03 09:45:21.621966 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:21.621814    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f9981fd-t4djq"]
Nov 03 09:45:21.624297 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:45:21.624275    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b2df33a_75de_45f9_b7c4_45f13ab82990.slice/crio-3bc0e3f634afdf0a34f7f2dbf59007ddd80fc8ee62a2785205921d27e72c383c WatchSource:0}: Error finding container 3bc0e3f634afdf0a34f7f2dbf59007ddd80fc8ee62a2785205921d27e72c383c: Status 404 returned error can't find the container with id 3bc0e3f634afdf0a34f7f2dbf59007ddd80fc8ee62a2785205921d27e72c383c
Nov 03 09:45:21.632478 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:21.632452    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894985-7kqh9"]
Nov 03 09:45:21.634953 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:21.634920    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3df1cc-29894985-7kqh9"]
Nov 03 09:45:21.870078 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:21.870052    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cd6af3f-fac0-471b-b368-5da14c939450" path="/var/lib/kubelet/pods/1cd6af3f-fac0-471b-b368-5da14c939450/volumes"
Nov 03 09:45:21.870433 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:21.870419    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ffb2aa7-f878-47e7-995e-f5db6f14b5a3" path="/var/lib/kubelet/pods/4ffb2aa7-f878-47e7-995e-f5db6f14b5a3/volumes"
Nov 03 09:45:21.870781 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:21.870769    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dfd9775-ced2-450a-8cc9-f4de56fafa52" path="/var/lib/kubelet/pods/7dfd9775-ced2-450a-8cc9-f4de56fafa52/volumes"
Nov 03 09:45:21.871444 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:21.871408    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2433454-f4b1-4109-a28c-5106db36e5ee" path="/var/lib/kubelet/pods/b2433454-f4b1-4109-a28c-5106db36e5ee/volumes"
Nov 03 09:45:21.871758 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:21.871746    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be2cbd18-00fc-42ff-84d8-1bed98edeb60" path="/var/lib/kubelet/pods/be2cbd18-00fc-42ff-84d8-1bed98edeb60/volumes"
Nov 03 09:45:21.872112 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:21.872100    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea1afde1-03bb-4db5-875f-43845656bf63" path="/var/lib/kubelet/pods/ea1afde1-03bb-4db5-875f-43845656bf63/volumes"
Nov 03 09:45:22.125623 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:22.125540    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-af514d-854f5df6d5-7xbnz" podUID="94aabaf0-2118-41f5-99b8-e558c95a5f46" containerName="payload-processing" containerID="cri-o://0894daf4cc73ae78455c4c59a88538cc94662404472a7043d0b1e19314992915" gracePeriod=25
Nov 03 09:45:22.233011 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:22.232970    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894985-wv4z2"]
Nov 03 09:45:22.323367 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:22.323334    4473 generic.go:356] "Generic (PLEG): container finished" podID="94aabaf0-2118-41f5-99b8-e558c95a5f46" containerID="0894daf4cc73ae78455c4c59a88538cc94662404472a7043d0b1e19314992915" exitCode=0
Nov 03 09:45:22.323778 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:22.323402    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-af514d-854f5df6d5-7xbnz" event={"ID":"94aabaf0-2118-41f5-99b8-e558c95a5f46","Type":"ContainerDied","Data":"0894daf4cc73ae78455c4c59a88538cc94662404472a7043d0b1e19314992915"}
Nov 03 09:45:22.325462 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:22.325439    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f9981fd-t4djq" event={"ID":"1b2df33a-75de-45f9-b7c4-45f13ab82990","Type":"ContainerStarted","Data":"db9f301444a0e60625c2e42f62c2d2be17e5eeca8d9e81a6805325d48cf20461"}
Nov 03 09:45:22.325586 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:22.325485    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f9981fd-t4djq" event={"ID":"1b2df33a-75de-45f9-b7c4-45f13ab82990","Type":"ContainerStarted","Data":"3bc0e3f634afdf0a34f7f2dbf59007ddd80fc8ee62a2785205921d27e72c383c"}
Nov 03 09:45:22.413298 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:22.413267    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894985-wv4z2"
Nov 03 09:45:22.416684 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:22.416662    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-af514d-854f5df6d5-7xbnz"
Nov 03 09:45:22.461085 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:22.461062    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94aabaf0-2118-41f5-99b8-e558c95a5f46-kube-api-access-5wkl4\" (UniqueName: \"kubernetes.io/projected/94aabaf0-2118-41f5-99b8-e558c95a5f46-kube-api-access-5wkl4\") pod \"94aabaf0-2118-41f5-99b8-e558c95a5f46\" (UID: \"94aabaf0-2118-41f5-99b8-e558c95a5f46\") "
Nov 03 09:45:22.461220 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:22.461095    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5-maas-api-service-ca\") pod \"a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5\" (UID: \"a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5\") "
Nov 03 09:45:22.461220 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:22.461128    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5-kube-api-access-bxn7s\" (UniqueName: \"kubernetes.io/projected/a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5-kube-api-access-bxn7s\") pod \"a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5\" (UID: \"a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5\") "
Nov 03 09:45:22.461220 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:22.461190    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94aabaf0-2118-41f5-99b8-e558c95a5f46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94aabaf0-2118-41f5-99b8-e558c95a5f46-plugins-config-volume\") pod \"94aabaf0-2118-41f5-99b8-e558c95a5f46\" (UID: \"94aabaf0-2118-41f5-99b8-e558c95a5f46\") "
Nov 03 09:45:22.461489 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:22.461468    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5-maas-api-service-ca" pod "a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5" (UID: "a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:45:22.461596 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:22.461573    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94aabaf0-2118-41f5-99b8-e558c95a5f46-plugins-config-volume" pod "94aabaf0-2118-41f5-99b8-e558c95a5f46" (UID: "94aabaf0-2118-41f5-99b8-e558c95a5f46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:45:22.461869 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:22.461850    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5-maas-api-service-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:45:22.461964 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:22.461875    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94aabaf0-2118-41f5-99b8-e558c95a5f46-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:45:22.463479 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:22.463460    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94aabaf0-2118-41f5-99b8-e558c95a5f46-kube-api-access-5wkl4" pod "94aabaf0-2118-41f5-99b8-e558c95a5f46" (UID: "94aabaf0-2118-41f5-99b8-e558c95a5f46"). InnerVolumeSpecName "kube-api-access-5wkl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:45:22.463554 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:22.463483    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5-kube-api-access-bxn7s" pod "a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5" (UID: "a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5"). InnerVolumeSpecName "kube-api-access-bxn7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:45:22.468771 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:22.468732    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f9981fd-t4djq" podStartSLOduration=2.468720551 podStartE2EDuration="2.468720551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:45:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:45:22.465362178 +0000 UTC m=+3946.734425652" watchObservedRunningTime="2026-11-03 09:45:22.468720551 +0000 UTC m=+3946.737783985"
Nov 03 09:45:22.562715 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:22.562692    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wkl4\" (UniqueName: \"kubernetes.io/projected/94aabaf0-2118-41f5-99b8-e558c95a5f46-kube-api-access-5wkl4\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:45:22.562856 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:22.562717    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxn7s\" (UniqueName: \"kubernetes.io/projected/a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5-kube-api-access-bxn7s\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:45:23.334069 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:23.333990    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-af514d-854f5df6d5-7xbnz"
Nov 03 09:45:23.334524 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:23.333990    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-af514d-854f5df6d5-7xbnz" event={"ID":"94aabaf0-2118-41f5-99b8-e558c95a5f46","Type":"ContainerDied","Data":"b93dba53bce225fbfefe2a7b66535f438260dfd70e9e4c9d54a678d50a9d9f5e"}
Nov 03 09:45:23.334524 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:23.334159    4473 scope.go:122] "RemoveContainer" containerID="0894daf4cc73ae78455c4c59a88538cc94662404472a7043d0b1e19314992915"
Nov 03 09:45:23.335644 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:23.335617    4473 generic.go:356] "Generic (PLEG): container finished" podID="1b2df33a-75de-45f9-b7c4-45f13ab82990" containerID="db9f301444a0e60625c2e42f62c2d2be17e5eeca8d9e81a6805325d48cf20461" exitCode=0
Nov 03 09:45:23.335727 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:23.335707    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f9981fd-t4djq" event={"ID":"1b2df33a-75de-45f9-b7c4-45f13ab82990","Type":"ContainerDied","Data":"db9f301444a0e60625c2e42f62c2d2be17e5eeca8d9e81a6805325d48cf20461"}
Nov 03 09:45:23.337245 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:23.337216    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894985-wv4z2" event={"ID":"a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5","Type":"ContainerDied","Data":"4767af0fb30392ebdb114db21e59fa1e5b0fa6c644b4faf567b29859bdc29088"}
Nov 03 09:45:23.337245 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:23.337246    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894985-wv4z2"
Nov 03 09:45:23.344974 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:23.344920    4473 scope.go:122] "RemoveContainer" containerID="0246199b10fb926681bda8235f892940852862b6fa3df054150dd8ad53177179"
Nov 03 09:45:23.512829 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:23.512791    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894985-wv4z2"]
Nov 03 09:45:23.527014 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:23.526985    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-465707-29894985-wv4z2"]
Nov 03 09:45:23.639245 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:23.639211    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-af514d-854f5df6d5-7xbnz"]
Nov 03 09:45:23.644185 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:23.644160    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-af514d-854f5df6d5-7xbnz"]
Nov 03 09:45:23.870180 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:23.870149    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94aabaf0-2118-41f5-99b8-e558c95a5f46" path="/var/lib/kubelet/pods/94aabaf0-2118-41f5-99b8-e558c95a5f46/volumes"
Nov 03 09:45:23.870573 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:23.870559    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5" path="/var/lib/kubelet/pods/a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5/volumes"
Nov 03 09:45:24.490473 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:24.490447    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f9981fd-t4djq"
Nov 03 09:45:24.585025 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:24.584999    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b2df33a-75de-45f9-b7c4-45f13ab82990-kube-api-access-9k28g\" (UniqueName: \"kubernetes.io/projected/1b2df33a-75de-45f9-b7c4-45f13ab82990-kube-api-access-9k28g\") pod \"1b2df33a-75de-45f9-b7c4-45f13ab82990\" (UID: \"1b2df33a-75de-45f9-b7c4-45f13ab82990\") "
Nov 03 09:45:24.585185 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:24.585045    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b2df33a-75de-45f9-b7c4-45f13ab82990-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b2df33a-75de-45f9-b7c4-45f13ab82990-maas-api-openshift-ca\") pod \"1b2df33a-75de-45f9-b7c4-45f13ab82990\" (UID: \"1b2df33a-75de-45f9-b7c4-45f13ab82990\") "
Nov 03 09:45:24.585185 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:24.585166    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b2df33a-75de-45f9-b7c4-45f13ab82990-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b2df33a-75de-45f9-b7c4-45f13ab82990-maas-api-xks-ca\") pod \"1b2df33a-75de-45f9-b7c4-45f13ab82990\" (UID: \"1b2df33a-75de-45f9-b7c4-45f13ab82990\") "
Nov 03 09:45:24.585414 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:24.585392    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b2df33a-75de-45f9-b7c4-45f13ab82990-maas-api-openshift-ca" pod "1b2df33a-75de-45f9-b7c4-45f13ab82990" (UID: "1b2df33a-75de-45f9-b7c4-45f13ab82990"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:45:24.585514 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:24.585499    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b2df33a-75de-45f9-b7c4-45f13ab82990-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:45:24.587311 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:24.587286    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b2df33a-75de-45f9-b7c4-45f13ab82990-maas-api-xks-ca" pod "1b2df33a-75de-45f9-b7c4-45f13ab82990" (UID: "1b2df33a-75de-45f9-b7c4-45f13ab82990"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:45:24.587428 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:24.587291    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b2df33a-75de-45f9-b7c4-45f13ab82990-kube-api-access-9k28g" pod "1b2df33a-75de-45f9-b7c4-45f13ab82990" (UID: "1b2df33a-75de-45f9-b7c4-45f13ab82990"). InnerVolumeSpecName "kube-api-access-9k28g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:45:24.686360 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:24.686332    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b2df33a-75de-45f9-b7c4-45f13ab82990-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:45:24.686509 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:24.686380    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k28g\" (UniqueName: \"kubernetes.io/projected/1b2df33a-75de-45f9-b7c4-45f13ab82990-kube-api-access-9k28g\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:45:25.352889 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:25.352860    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f9981fd-t4djq"
Nov 03 09:45:25.353101 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:25.352890    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f9981fd-t4djq" event={"ID":"1b2df33a-75de-45f9-b7c4-45f13ab82990","Type":"ContainerDied","Data":"3bc0e3f634afdf0a34f7f2dbf59007ddd80fc8ee62a2785205921d27e72c383c"}
Nov 03 09:45:25.353101 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:25.352939    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bc0e3f634afdf0a34f7f2dbf59007ddd80fc8ee62a2785205921d27e72c383c"
Nov 03 09:45:38.723163 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:45:38.723129    4473 scope.go:122] "RemoveContainer" containerID="4bd74d9920f6769412c06986d427f54026477900a8cd24dc5ced195dcb234b1a"
Nov 03 09:45:46.938291 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:45:46.938252    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:46:10.104530 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:10.104494    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:46:16.937110 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:46:16.937027    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:46:35.600643 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.600611    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-627e5f/sub-test-model-627e5f-kserve-6785c9fd7f-z4chr"]
Nov 03 09:46:35.601116 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.601107    4473 cpu_manager.go:414] "containerMap: removing container" podUID="ea1afde1-03bb-4db5-875f-43845656bf63" containerName="cleanup"
Nov 03 09:46:35.601163 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.601119    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea1afde1-03bb-4db5-875f-43845656bf63" containerName="cleanup"
Nov 03 09:46:35.601163 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.601135    4473 cpu_manager.go:414] "containerMap: removing container" podUID="a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5" containerName="cleanup"
Nov 03 09:46:35.601163 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.601140    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5" containerName="cleanup"
Nov 03 09:46:35.601163 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.601154    4473 cpu_manager.go:414] "containerMap: removing container" podUID="1b2df33a-75de-45f9-b7c4-45f13ab82990" containerName="revoke-keys"
Nov 03 09:46:35.601163 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.601160    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b2df33a-75de-45f9-b7c4-45f13ab82990" containerName="revoke-keys"
Nov 03 09:46:35.601326 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.601174    4473 cpu_manager.go:414] "containerMap: removing container" podUID="94aabaf0-2118-41f5-99b8-e558c95a5f46" containerName="payload-processing"
Nov 03 09:46:35.601326 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.601179    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94aabaf0-2118-41f5-99b8-e558c95a5f46" containerName="payload-processing"
Nov 03 09:46:35.601326 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.601189    4473 cpu_manager.go:414] "containerMap: removing container" podUID="a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5" containerName="cleanup"
Nov 03 09:46:35.601326 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.601193    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5" containerName="cleanup"
Nov 03 09:46:35.601326 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.601208    4473 cpu_manager.go:414] "containerMap: removing container" podUID="a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5" containerName="cleanup"
Nov 03 09:46:35.601326 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.601212    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5" containerName="cleanup"
Nov 03 09:46:35.601326 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.601225    4473 cpu_manager.go:414] "containerMap: removing container" podUID="7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed" containerName="cleanup"
Nov 03 09:46:35.601326 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.601229    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed" containerName="cleanup"
Nov 03 09:46:35.601326 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.601235    4473 cpu_manager.go:414] "containerMap: removing container" podUID="1cd6af3f-fac0-471b-b368-5da14c939450" containerName="cleanup"
Nov 03 09:46:35.601326 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.601239    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cd6af3f-fac0-471b-b368-5da14c939450" containerName="cleanup"
Nov 03 09:46:35.601326 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.601309    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2433454-f4b1-4109-a28c-5106db36e5ee" containerName="cleanup"
Nov 03 09:46:35.601326 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.601318    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e39b5ea-acdb-4a9e-bc1b-bdf9bd4553ed" containerName="cleanup"
Nov 03 09:46:35.601326 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.601325    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cd6af3f-fac0-471b-b368-5da14c939450" containerName="cleanup"
Nov 03 09:46:35.601764 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.601332    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5" containerName="cleanup"
Nov 03 09:46:35.601764 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.601339    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b2df33a-75de-45f9-b7c4-45f13ab82990" containerName="revoke-keys"
Nov 03 09:46:35.601764 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.601347    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="94aabaf0-2118-41f5-99b8-e558c95a5f46" containerName="payload-processing"
Nov 03 09:46:35.601764 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.601353    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5" containerName="cleanup"
Nov 03 09:46:35.601764 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.601358    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ffb2aa7-f878-47e7-995e-f5db6f14b5a3" containerName="cleanup"
Nov 03 09:46:35.604588 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.604573    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-627e5f/sub-test-model-627e5f-kserve-6785c9fd7f-z4chr"
Nov 03 09:46:35.663755 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.663723    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bf62832-a048-47e7-91d5-cb4eb34e7e26-home\") pod \"sub-test-model-627e5f-kserve-6785c9fd7f-z4chr\" (UID: \"9bf62832-a048-47e7-91d5-cb4eb34e7e26\") " pod="ai-tenant-e2e-shared-a-w3-627e5f/sub-test-model-627e5f-kserve-6785c9fd7f-z4chr"
Nov 03 09:46:35.663963 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.663763    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bf62832-a048-47e7-91d5-cb4eb34e7e26-dshm\") pod \"sub-test-model-627e5f-kserve-6785c9fd7f-z4chr\" (UID: \"9bf62832-a048-47e7-91d5-cb4eb34e7e26\") " pod="ai-tenant-e2e-shared-a-w3-627e5f/sub-test-model-627e5f-kserve-6785c9fd7f-z4chr"
Nov 03 09:46:35.663963 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.663796    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szqxc\" (UniqueName: \"kubernetes.io/projected/9bf62832-a048-47e7-91d5-cb4eb34e7e26-kube-api-access-szqxc\") pod \"sub-test-model-627e5f-kserve-6785c9fd7f-z4chr\" (UID: \"9bf62832-a048-47e7-91d5-cb4eb34e7e26\") " pod="ai-tenant-e2e-shared-a-w3-627e5f/sub-test-model-627e5f-kserve-6785c9fd7f-z4chr"
Nov 03 09:46:35.664077 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.663959    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bf62832-a048-47e7-91d5-cb4eb34e7e26-tmp-dir\") pod \"sub-test-model-627e5f-kserve-6785c9fd7f-z4chr\" (UID: \"9bf62832-a048-47e7-91d5-cb4eb34e7e26\") " pod="ai-tenant-e2e-shared-a-w3-627e5f/sub-test-model-627e5f-kserve-6785c9fd7f-z4chr"
Nov 03 09:46:35.664077 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.664048    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bf62832-a048-47e7-91d5-cb4eb34e7e26-model-cache\") pod \"sub-test-model-627e5f-kserve-6785c9fd7f-z4chr\" (UID: \"9bf62832-a048-47e7-91d5-cb4eb34e7e26\") " pod="ai-tenant-e2e-shared-a-w3-627e5f/sub-test-model-627e5f-kserve-6785c9fd7f-z4chr"
Nov 03 09:46:35.664147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.664097    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bf62832-a048-47e7-91d5-cb4eb34e7e26-tls-certs\") pod \"sub-test-model-627e5f-kserve-6785c9fd7f-z4chr\" (UID: \"9bf62832-a048-47e7-91d5-cb4eb34e7e26\") " pod="ai-tenant-e2e-shared-a-w3-627e5f/sub-test-model-627e5f-kserve-6785c9fd7f-z4chr"
Nov 03 09:46:35.708739 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.708689    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-627e5f\"/\"openshift-service-ca.crt\""
Nov 03 09:46:35.708899 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.708813    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-627e5f\"/\"default-dockercfg-jb48d\""
Nov 03 09:46:35.709530 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.709504    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-627e5f\"/\"kube-root-ca.crt\""
Nov 03 09:46:35.709700 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.709505    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-627e5f\"/\"sub-test-model-627e5f-kserve-self-signed-certs\""
Nov 03 09:46:35.715031 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.715005    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-627e5f/sub-test-model-627e5f-kserve-6785c9fd7f-z4chr"]
Nov 03 09:46:35.777002 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.776931    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bf62832-a048-47e7-91d5-cb4eb34e7e26-home\") pod \"sub-test-model-627e5f-kserve-6785c9fd7f-z4chr\" (UID: \"9bf62832-a048-47e7-91d5-cb4eb34e7e26\") " pod="ai-tenant-e2e-shared-a-w3-627e5f/sub-test-model-627e5f-kserve-6785c9fd7f-z4chr"
Nov 03 09:46:35.777205 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.777015    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bf62832-a048-47e7-91d5-cb4eb34e7e26-dshm\") pod \"sub-test-model-627e5f-kserve-6785c9fd7f-z4chr\" (UID: \"9bf62832-a048-47e7-91d5-cb4eb34e7e26\") " pod="ai-tenant-e2e-shared-a-w3-627e5f/sub-test-model-627e5f-kserve-6785c9fd7f-z4chr"
Nov 03 09:46:35.777205 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.777093    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szqxc\" (UniqueName: \"kubernetes.io/projected/9bf62832-a048-47e7-91d5-cb4eb34e7e26-kube-api-access-szqxc\") pod \"sub-test-model-627e5f-kserve-6785c9fd7f-z4chr\" (UID: \"9bf62832-a048-47e7-91d5-cb4eb34e7e26\") " pod="ai-tenant-e2e-shared-a-w3-627e5f/sub-test-model-627e5f-kserve-6785c9fd7f-z4chr"
Nov 03 09:46:35.777205 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.777176    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bf62832-a048-47e7-91d5-cb4eb34e7e26-tmp-dir\") pod \"sub-test-model-627e5f-kserve-6785c9fd7f-z4chr\" (UID: \"9bf62832-a048-47e7-91d5-cb4eb34e7e26\") " pod="ai-tenant-e2e-shared-a-w3-627e5f/sub-test-model-627e5f-kserve-6785c9fd7f-z4chr"
Nov 03 09:46:35.777383 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.777232    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bf62832-a048-47e7-91d5-cb4eb34e7e26-model-cache\") pod \"sub-test-model-627e5f-kserve-6785c9fd7f-z4chr\" (UID: \"9bf62832-a048-47e7-91d5-cb4eb34e7e26\") " pod="ai-tenant-e2e-shared-a-w3-627e5f/sub-test-model-627e5f-kserve-6785c9fd7f-z4chr"
Nov 03 09:46:35.777383 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.777311    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bf62832-a048-47e7-91d5-cb4eb34e7e26-tls-certs\") pod \"sub-test-model-627e5f-kserve-6785c9fd7f-z4chr\" (UID: \"9bf62832-a048-47e7-91d5-cb4eb34e7e26\") " pod="ai-tenant-e2e-shared-a-w3-627e5f/sub-test-model-627e5f-kserve-6785c9fd7f-z4chr"
Nov 03 09:46:35.778642 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.778547    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bf62832-a048-47e7-91d5-cb4eb34e7e26-home\") pod \"sub-test-model-627e5f-kserve-6785c9fd7f-z4chr\" (UID: \"9bf62832-a048-47e7-91d5-cb4eb34e7e26\") " pod="ai-tenant-e2e-shared-a-w3-627e5f/sub-test-model-627e5f-kserve-6785c9fd7f-z4chr"
Nov 03 09:46:35.779053 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.778681    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bf62832-a048-47e7-91d5-cb4eb34e7e26-tmp-dir\") pod \"sub-test-model-627e5f-kserve-6785c9fd7f-z4chr\" (UID: \"9bf62832-a048-47e7-91d5-cb4eb34e7e26\") " pod="ai-tenant-e2e-shared-a-w3-627e5f/sub-test-model-627e5f-kserve-6785c9fd7f-z4chr"
Nov 03 09:46:35.779053 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.778925    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bf62832-a048-47e7-91d5-cb4eb34e7e26-model-cache\") pod \"sub-test-model-627e5f-kserve-6785c9fd7f-z4chr\" (UID: \"9bf62832-a048-47e7-91d5-cb4eb34e7e26\") " pod="ai-tenant-e2e-shared-a-w3-627e5f/sub-test-model-627e5f-kserve-6785c9fd7f-z4chr"
Nov 03 09:46:35.783008 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.782986    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bf62832-a048-47e7-91d5-cb4eb34e7e26-dshm\") pod \"sub-test-model-627e5f-kserve-6785c9fd7f-z4chr\" (UID: \"9bf62832-a048-47e7-91d5-cb4eb34e7e26\") " pod="ai-tenant-e2e-shared-a-w3-627e5f/sub-test-model-627e5f-kserve-6785c9fd7f-z4chr"
Nov 03 09:46:35.783442 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.783423    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bf62832-a048-47e7-91d5-cb4eb34e7e26-tls-certs\") pod \"sub-test-model-627e5f-kserve-6785c9fd7f-z4chr\" (UID: \"9bf62832-a048-47e7-91d5-cb4eb34e7e26\") " pod="ai-tenant-e2e-shared-a-w3-627e5f/sub-test-model-627e5f-kserve-6785c9fd7f-z4chr"
Nov 03 09:46:35.837051 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.837019    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szqxc\" (UniqueName: \"kubernetes.io/projected/9bf62832-a048-47e7-91d5-cb4eb34e7e26-kube-api-access-szqxc\") pod \"sub-test-model-627e5f-kserve-6785c9fd7f-z4chr\" (UID: \"9bf62832-a048-47e7-91d5-cb4eb34e7e26\") " pod="ai-tenant-e2e-shared-a-w3-627e5f/sub-test-model-627e5f-kserve-6785c9fd7f-z4chr"
Nov 03 09:46:35.916839 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:35.916794    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-627e5f/sub-test-model-627e5f-kserve-6785c9fd7f-z4chr"
Nov 03 09:46:36.316178 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:36.316155    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-627e5f/sub-test-model-627e5f-kserve-6785c9fd7f-z4chr"]
Nov 03 09:46:36.317462 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:46:36.317433    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9bf62832_a048_47e7_91d5_cb4eb34e7e26.slice/crio-576a97e7c68986e302c383853bf4f99cab8ca683dcebabb8939968f0b5cf450f WatchSource:0}: Error finding container 576a97e7c68986e302c383853bf4f99cab8ca683dcebabb8939968f0b5cf450f: Status 404 returned error can't find the container with id 576a97e7c68986e302c383853bf4f99cab8ca683dcebabb8939968f0b5cf450f
Nov 03 09:46:36.796594 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:36.796562    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-627e5f/sub-test-model-627e5f-kserve-6785c9fd7f-z4chr" event={"ID":"9bf62832-a048-47e7-91d5-cb4eb34e7e26","Type":"ContainerStarted","Data":"72890e92d941f1a7147c0a2b7f5fc300bc442c8200fe08b9b4fe9b5bb464ae24"}
Nov 03 09:46:36.796995 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:36.796603    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-627e5f/sub-test-model-627e5f-kserve-6785c9fd7f-z4chr" event={"ID":"9bf62832-a048-47e7-91d5-cb4eb34e7e26","Type":"ContainerStarted","Data":"576a97e7c68986e302c383853bf4f99cab8ca683dcebabb8939968f0b5cf450f"}
Nov 03 09:46:36.914105 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:36.914055    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-627e5f/sub-test-model-627e5f-kserve-6785c9fd7f-z4chr" podStartSLOduration=1.9140418160000001 podStartE2EDuration="1.914041816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:46:36.912504298 +0000 UTC m=+4021.181567754" watchObservedRunningTime="2026-11-03 09:46:36.914041816 +0000 UTC m=+4021.183105299"
Nov 03 09:46:36.917636 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:36.917613    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-627e5f/sub-test-model-627e5f-kserve-6785c9fd7f-z4chr"
Nov 03 09:46:45.918013 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:45.917962    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-627e5f/sub-test-model-627e5f-kserve-6785c9fd7f-z4chr"
Nov 03 09:46:45.930897 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:45.930870    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-627e5f/sub-test-model-627e5f-kserve-6785c9fd7f-z4chr"
Nov 03 09:46:46.863570 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:46.863539    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-627e5f/sub-test-model-627e5f-kserve-6785c9fd7f-z4chr"
Nov 03 09:46:46.937350 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:46:46.937318    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:46:57.712328 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:46:57.712288    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:47:02.428166 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:02.428130    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f6de690-wc92p"]
Nov 03 09:47:02.428826 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:02.428809    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4730ab7-f5a6-4e3f-9ad8-ba020c5a48a5" containerName="cleanup"
Nov 03 09:47:02.432140 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:02.432125    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f6de690-wc92p"
Nov 03 09:47:02.490213 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:02.490186    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f6de690-wc92p"]
Nov 03 09:47:02.536177 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:02.536148    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:47:02.536177 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:02.536144    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:47:02.536177 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:02.536176    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:47:02.536430 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:02.536227    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:47:02.607781 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:02.607749    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448e793c-ce4a-4210-a42f-6b1ec2acbeb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f6de690-wc92p\" (UID: \"448e793c-ce4a-4210-a42f-6b1ec2acbeb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f6de690-wc92p"
Nov 03 09:47:02.607781 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:02.607783    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dv92\" (UniqueName: \"kubernetes.io/projected/448e793c-ce4a-4210-a42f-6b1ec2acbeb9-kube-api-access-9dv92\") pod \"maas-api-revoke-sub-2f6de690-wc92p\" (UID: \"448e793c-ce4a-4210-a42f-6b1ec2acbeb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f6de690-wc92p"
Nov 03 09:47:02.608033 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:02.607934    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448e793c-ce4a-4210-a42f-6b1ec2acbeb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f6de690-wc92p\" (UID: \"448e793c-ce4a-4210-a42f-6b1ec2acbeb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f6de690-wc92p"
Nov 03 09:47:02.709114 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:02.709045    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448e793c-ce4a-4210-a42f-6b1ec2acbeb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f6de690-wc92p\" (UID: \"448e793c-ce4a-4210-a42f-6b1ec2acbeb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f6de690-wc92p"
Nov 03 09:47:02.709114 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:02.709077    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dv92\" (UniqueName: \"kubernetes.io/projected/448e793c-ce4a-4210-a42f-6b1ec2acbeb9-kube-api-access-9dv92\") pod \"maas-api-revoke-sub-2f6de690-wc92p\" (UID: \"448e793c-ce4a-4210-a42f-6b1ec2acbeb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f6de690-wc92p"
Nov 03 09:47:02.709372 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:02.709162    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448e793c-ce4a-4210-a42f-6b1ec2acbeb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f6de690-wc92p\" (UID: \"448e793c-ce4a-4210-a42f-6b1ec2acbeb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f6de690-wc92p"
Nov 03 09:47:02.709826 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:02.709806    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448e793c-ce4a-4210-a42f-6b1ec2acbeb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f6de690-wc92p\" (UID: \"448e793c-ce4a-4210-a42f-6b1ec2acbeb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f6de690-wc92p"
Nov 03 09:47:02.711573 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:02.711547    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448e793c-ce4a-4210-a42f-6b1ec2acbeb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f6de690-wc92p\" (UID: \"448e793c-ce4a-4210-a42f-6b1ec2acbeb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f6de690-wc92p"
Nov 03 09:47:02.768223 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:02.768198    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dv92\" (UniqueName: \"kubernetes.io/projected/448e793c-ce4a-4210-a42f-6b1ec2acbeb9-kube-api-access-9dv92\") pod \"maas-api-revoke-sub-2f6de690-wc92p\" (UID: \"448e793c-ce4a-4210-a42f-6b1ec2acbeb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f6de690-wc92p"
Nov 03 09:47:03.046196 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:03.046115    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f6de690-wc92p"
Nov 03 09:47:03.422021 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:03.421998    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f6de690-wc92p"]
Nov 03 09:47:03.424286 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:47:03.424259    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod448e793c_ce4a_4210_a42f_6b1ec2acbeb9.slice/crio-1ef0ec026fc6c99050db69dab6908749a3668cc5df8d4cc860f7d50c66eec62d WatchSource:0}: Error finding container 1ef0ec026fc6c99050db69dab6908749a3668cc5df8d4cc860f7d50c66eec62d: Status 404 returned error can't find the container with id 1ef0ec026fc6c99050db69dab6908749a3668cc5df8d4cc860f7d50c66eec62d
Nov 03 09:47:03.966516 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:03.966481    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f6de690-wc92p" event={"ID":"448e793c-ce4a-4210-a42f-6b1ec2acbeb9","Type":"ContainerStarted","Data":"fb0d377bc6346095de61daf082dcb7aaab33118f7ab208deada2574f2cd39aa9"}
Nov 03 09:47:03.966958 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:03.966539    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f6de690-wc92p" event={"ID":"448e793c-ce4a-4210-a42f-6b1ec2acbeb9","Type":"ContainerStarted","Data":"1ef0ec026fc6c99050db69dab6908749a3668cc5df8d4cc860f7d50c66eec62d"}
Nov 03 09:47:04.085221 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:04.085114    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f6de690-wc92p" podStartSLOduration=2.085098606 podStartE2EDuration="2.085098606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:47:04.083577856 +0000 UTC m=+4048.352641302" watchObservedRunningTime="2026-11-03 09:47:04.085098606 +0000 UTC m=+4048.354162040"
Nov 03 09:47:04.974137 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:04.974103    4473 generic.go:356] "Generic (PLEG): container finished" podID="448e793c-ce4a-4210-a42f-6b1ec2acbeb9" containerID="fb0d377bc6346095de61daf082dcb7aaab33118f7ab208deada2574f2cd39aa9" exitCode=0
Nov 03 09:47:04.974608 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:04.974194    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f6de690-wc92p" event={"ID":"448e793c-ce4a-4210-a42f-6b1ec2acbeb9","Type":"ContainerDied","Data":"fb0d377bc6346095de61daf082dcb7aaab33118f7ab208deada2574f2cd39aa9"}
Nov 03 09:47:06.124506 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:06.124475    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f6de690-wc92p"
Nov 03 09:47:06.146833 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:06.146799    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/448e793c-ce4a-4210-a42f-6b1ec2acbeb9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448e793c-ce4a-4210-a42f-6b1ec2acbeb9-maas-api-xks-ca\") pod \"448e793c-ce4a-4210-a42f-6b1ec2acbeb9\" (UID: \"448e793c-ce4a-4210-a42f-6b1ec2acbeb9\") "
Nov 03 09:47:06.147033 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:06.146921    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/448e793c-ce4a-4210-a42f-6b1ec2acbeb9-kube-api-access-9dv92\" (UniqueName: \"kubernetes.io/projected/448e793c-ce4a-4210-a42f-6b1ec2acbeb9-kube-api-access-9dv92\") pod \"448e793c-ce4a-4210-a42f-6b1ec2acbeb9\" (UID: \"448e793c-ce4a-4210-a42f-6b1ec2acbeb9\") "
Nov 03 09:47:06.147092 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:06.147039    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/448e793c-ce4a-4210-a42f-6b1ec2acbeb9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448e793c-ce4a-4210-a42f-6b1ec2acbeb9-maas-api-openshift-ca\") pod \"448e793c-ce4a-4210-a42f-6b1ec2acbeb9\" (UID: \"448e793c-ce4a-4210-a42f-6b1ec2acbeb9\") "
Nov 03 09:47:06.147367 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:06.147344    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/448e793c-ce4a-4210-a42f-6b1ec2acbeb9-maas-api-openshift-ca" pod "448e793c-ce4a-4210-a42f-6b1ec2acbeb9" (UID: "448e793c-ce4a-4210-a42f-6b1ec2acbeb9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:47:06.147628 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:06.147551    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448e793c-ce4a-4210-a42f-6b1ec2acbeb9-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:47:06.148995 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:06.148969    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/448e793c-ce4a-4210-a42f-6b1ec2acbeb9-maas-api-xks-ca" pod "448e793c-ce4a-4210-a42f-6b1ec2acbeb9" (UID: "448e793c-ce4a-4210-a42f-6b1ec2acbeb9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:47:06.149202 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:06.149174    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/448e793c-ce4a-4210-a42f-6b1ec2acbeb9-kube-api-access-9dv92" pod "448e793c-ce4a-4210-a42f-6b1ec2acbeb9" (UID: "448e793c-ce4a-4210-a42f-6b1ec2acbeb9"). InnerVolumeSpecName "kube-api-access-9dv92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:47:06.248700 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:06.248610    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dv92\" (UniqueName: \"kubernetes.io/projected/448e793c-ce4a-4210-a42f-6b1ec2acbeb9-kube-api-access-9dv92\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:47:06.248700 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:06.248638    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448e793c-ce4a-4210-a42f-6b1ec2acbeb9-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:47:06.987277 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:06.987241    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f6de690-wc92p" event={"ID":"448e793c-ce4a-4210-a42f-6b1ec2acbeb9","Type":"ContainerDied","Data":"1ef0ec026fc6c99050db69dab6908749a3668cc5df8d4cc860f7d50c66eec62d"}
Nov 03 09:47:06.987565 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:06.987522    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ef0ec026fc6c99050db69dab6908749a3668cc5df8d4cc860f7d50c66eec62d"
Nov 03 09:47:06.987565 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:06.987273    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f6de690-wc92p"
Nov 03 09:47:16.937772 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:47:16.937734    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:47:43.838188 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:43.838155    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-b196c8/sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77"]
Nov 03 09:47:43.838688 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:43.838679    4473 cpu_manager.go:414] "containerMap: removing container" podUID="448e793c-ce4a-4210-a42f-6b1ec2acbeb9" containerName="revoke-keys"
Nov 03 09:47:43.838732 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:43.838691    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="448e793c-ce4a-4210-a42f-6b1ec2acbeb9" containerName="revoke-keys"
Nov 03 09:47:43.838788 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:43.838777    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="448e793c-ce4a-4210-a42f-6b1ec2acbeb9" containerName="revoke-keys"
Nov 03 09:47:43.841753 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:43.841735    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b196c8/sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77"
Nov 03 09:47:43.933663 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:43.933630    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5dce140-fcd3-420f-bb8e-f0cb8562d047-dshm\") pod \"sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77\" (UID: \"a5dce140-fcd3-420f-bb8e-f0cb8562d047\") " pod="ai-tenant-e2e-shared-b-w3-b196c8/sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77"
Nov 03 09:47:43.933861 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:43.933738    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5dce140-fcd3-420f-bb8e-f0cb8562d047-tls-certs\") pod \"sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77\" (UID: \"a5dce140-fcd3-420f-bb8e-f0cb8562d047\") " pod="ai-tenant-e2e-shared-b-w3-b196c8/sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77"
Nov 03 09:47:43.933861 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:43.933830    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5dce140-fcd3-420f-bb8e-f0cb8562d047-tmp-dir\") pod \"sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77\" (UID: \"a5dce140-fcd3-420f-bb8e-f0cb8562d047\") " pod="ai-tenant-e2e-shared-b-w3-b196c8/sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77"
Nov 03 09:47:43.934047 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:43.933915    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5dce140-fcd3-420f-bb8e-f0cb8562d047-home\") pod \"sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77\" (UID: \"a5dce140-fcd3-420f-bb8e-f0cb8562d047\") " pod="ai-tenant-e2e-shared-b-w3-b196c8/sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77"
Nov 03 09:47:43.934047 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:43.933996    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5dce140-fcd3-420f-bb8e-f0cb8562d047-model-cache\") pod \"sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77\" (UID: \"a5dce140-fcd3-420f-bb8e-f0cb8562d047\") " pod="ai-tenant-e2e-shared-b-w3-b196c8/sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77"
Nov 03 09:47:43.934171 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:43.934059    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtwrt\" (UniqueName: \"kubernetes.io/projected/a5dce140-fcd3-420f-bb8e-f0cb8562d047-kube-api-access-qtwrt\") pod \"sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77\" (UID: \"a5dce140-fcd3-420f-bb8e-f0cb8562d047\") " pod="ai-tenant-e2e-shared-b-w3-b196c8/sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77"
Nov 03 09:47:43.946238 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:43.946210    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b196c8\"/\"kube-root-ca.crt\""
Nov 03 09:47:43.946914 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:43.946900    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b196c8\"/\"sub-test-model-b196c8-kserve-self-signed-certs\""
Nov 03 09:47:43.947018 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:43.947004    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b196c8\"/\"openshift-service-ca.crt\""
Nov 03 09:47:43.947083 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:43.947042    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b196c8\"/\"default-dockercfg-99khw\""
Nov 03 09:47:43.950564 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:43.950543    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b196c8/sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77"]
Nov 03 09:47:44.035427 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:44.035397    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5dce140-fcd3-420f-bb8e-f0cb8562d047-dshm\") pod \"sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77\" (UID: \"a5dce140-fcd3-420f-bb8e-f0cb8562d047\") " pod="ai-tenant-e2e-shared-b-w3-b196c8/sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77"
Nov 03 09:47:44.035652 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:44.035464    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5dce140-fcd3-420f-bb8e-f0cb8562d047-tls-certs\") pod \"sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77\" (UID: \"a5dce140-fcd3-420f-bb8e-f0cb8562d047\") " pod="ai-tenant-e2e-shared-b-w3-b196c8/sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77"
Nov 03 09:47:44.035652 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:44.035513    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5dce140-fcd3-420f-bb8e-f0cb8562d047-tmp-dir\") pod \"sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77\" (UID: \"a5dce140-fcd3-420f-bb8e-f0cb8562d047\") " pod="ai-tenant-e2e-shared-b-w3-b196c8/sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77"
Nov 03 09:47:44.035652 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:44.035589    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5dce140-fcd3-420f-bb8e-f0cb8562d047-home\") pod \"sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77\" (UID: \"a5dce140-fcd3-420f-bb8e-f0cb8562d047\") " pod="ai-tenant-e2e-shared-b-w3-b196c8/sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77"
Nov 03 09:47:44.035652 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:44.035625    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5dce140-fcd3-420f-bb8e-f0cb8562d047-model-cache\") pod \"sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77\" (UID: \"a5dce140-fcd3-420f-bb8e-f0cb8562d047\") " pod="ai-tenant-e2e-shared-b-w3-b196c8/sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77"
Nov 03 09:47:44.035881 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:44.035673    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtwrt\" (UniqueName: \"kubernetes.io/projected/a5dce140-fcd3-420f-bb8e-f0cb8562d047-kube-api-access-qtwrt\") pod \"sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77\" (UID: \"a5dce140-fcd3-420f-bb8e-f0cb8562d047\") " pod="ai-tenant-e2e-shared-b-w3-b196c8/sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77"
Nov 03 09:47:44.036018 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:44.035957    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5dce140-fcd3-420f-bb8e-f0cb8562d047-home\") pod \"sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77\" (UID: \"a5dce140-fcd3-420f-bb8e-f0cb8562d047\") " pod="ai-tenant-e2e-shared-b-w3-b196c8/sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77"
Nov 03 09:47:44.036018 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:44.035997    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5dce140-fcd3-420f-bb8e-f0cb8562d047-tmp-dir\") pod \"sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77\" (UID: \"a5dce140-fcd3-420f-bb8e-f0cb8562d047\") " pod="ai-tenant-e2e-shared-b-w3-b196c8/sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77"
Nov 03 09:47:44.036437 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:44.036415    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5dce140-fcd3-420f-bb8e-f0cb8562d047-model-cache\") pod \"sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77\" (UID: \"a5dce140-fcd3-420f-bb8e-f0cb8562d047\") " pod="ai-tenant-e2e-shared-b-w3-b196c8/sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77"
Nov 03 09:47:44.037980 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:44.037932    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5dce140-fcd3-420f-bb8e-f0cb8562d047-dshm\") pod \"sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77\" (UID: \"a5dce140-fcd3-420f-bb8e-f0cb8562d047\") " pod="ai-tenant-e2e-shared-b-w3-b196c8/sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77"
Nov 03 09:47:44.038116 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:44.038100    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5dce140-fcd3-420f-bb8e-f0cb8562d047-tls-certs\") pod \"sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77\" (UID: \"a5dce140-fcd3-420f-bb8e-f0cb8562d047\") " pod="ai-tenant-e2e-shared-b-w3-b196c8/sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77"
Nov 03 09:47:44.096907 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:44.096819    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtwrt\" (UniqueName: \"kubernetes.io/projected/a5dce140-fcd3-420f-bb8e-f0cb8562d047-kube-api-access-qtwrt\") pod \"sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77\" (UID: \"a5dce140-fcd3-420f-bb8e-f0cb8562d047\") " pod="ai-tenant-e2e-shared-b-w3-b196c8/sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77"
Nov 03 09:47:44.153769 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:44.153731    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b196c8/sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77"
Nov 03 09:47:44.536324 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:44.536293    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b196c8/sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77"]
Nov 03 09:47:44.537672 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:47:44.537646    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5dce140_fcd3_420f_bb8e_f0cb8562d047.slice/crio-8c2655b94bd665d95296b2b9a091eb077c3e388f71bcfe71a3a6c85a51fb8516 WatchSource:0}: Error finding container 8c2655b94bd665d95296b2b9a091eb077c3e388f71bcfe71a3a6c85a51fb8516: Status 404 returned error can't find the container with id 8c2655b94bd665d95296b2b9a091eb077c3e388f71bcfe71a3a6c85a51fb8516
Nov 03 09:47:45.231876 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:45.231832    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b196c8/sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77" event={"ID":"a5dce140-fcd3-420f-bb8e-f0cb8562d047","Type":"ContainerStarted","Data":"664e2201cfd2f8afc59ef5820a0cc61f91818fee8d2cebd348b62ff9f6235547"}
Nov 03 09:47:45.232299 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:45.231888    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b196c8/sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77" event={"ID":"a5dce140-fcd3-420f-bb8e-f0cb8562d047","Type":"ContainerStarted","Data":"8c2655b94bd665d95296b2b9a091eb077c3e388f71bcfe71a3a6c85a51fb8516"}
Nov 03 09:47:45.351969 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:45.351907    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-b196c8/sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77" podStartSLOduration=2.3518914459999998 podStartE2EDuration="2.351891446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:47:45.348509373 +0000 UTC m=+4089.617572821" watchObservedRunningTime="2026-11-03 09:47:45.351891446 +0000 UTC m=+4089.620954894"
Nov 03 09:47:46.154131 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:46.154097    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-b196c8/sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77"
Nov 03 09:47:46.937050 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:47:46.936975    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:47:53.734887 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:53.734851    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:47:54.153999 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:54.153967    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-b196c8/sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77"
Nov 03 09:47:54.166648 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:54.166622    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-b196c8/sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77"
Nov 03 09:47:54.299821 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:47:54.299790    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-b196c8/sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77"
Nov 03 09:48:16.937727 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:48:16.937678    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:48:46.937470 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:48:46.937434    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:48:50.826052 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:48:50.826018    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:48:59.062496 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:48:59.062456    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcb36718-ql7dd"]
Nov 03 09:48:59.112026 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:48:59.112000    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcb36718-ql7dd"]
Nov 03 09:48:59.871206 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:48:59.871176    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45f126fe-f88c-43af-8c53-a24b0f891c25" path="/var/lib/kubelet/pods/45f126fe-f88c-43af-8c53-a24b0f891c25/volumes"
Nov 03 09:49:11.053188 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:11.053151    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2646e5c6-7kzw9"]
Nov 03 09:49:11.103667 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:11.103640    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2646e5c6-7kzw9"]
Nov 03 09:49:11.870027 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:11.869995    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6958024a-dbcc-4d7f-80b9-c8ec8314ea52" path="/var/lib/kubelet/pods/6958024a-dbcc-4d7f-80b9-c8ec8314ea52/volumes"
Nov 03 09:49:16.938504 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:49:16.938426    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:49:36.801187 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:36.801157    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 09:49:36.804150 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:36.804128    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 09:49:38.198094 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:38.198063    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:49:41.607433 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:41.607397    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-3df1cc-6d58d67765-44gqp"]
Nov 03 09:49:46.607739 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:46.607663    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3df1cc-6d58d67765-44gqp" podUID="9f2cb2de-df5b-4f79-95f1-4232463e82bb" containerName="payload-processing" containerID="cri-o://9b04779f0939659e793305130e31d8046b84cd548035b00bb776d99cb6770ec1" gracePeriod=25
Nov 03 09:49:46.937341 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:49:46.937310    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:49:48.683579 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:48.683536    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3df1cc-6d58d67765-44gqp" podUID="9f2cb2de-df5b-4f79-95f1-4232463e82bb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:49:55.792636 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:55.792605    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f8439f5b-58b76f4b8c-x2dl8"]
Nov 03 09:49:55.796422 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:55.796406    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8439f5b-58b76f4b8c-x2dl8"
Nov 03 09:49:55.855528 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:55.855496    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f8439f5b-58b76f4b8c-x2dl8"]
Nov 03 09:49:55.902539 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:55.902494    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f8439f5b-dockercfg-w984q\""
Nov 03 09:49:55.903573 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:55.903048    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f8439f5b\""
Nov 03 09:49:55.940591 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:55.940552    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pgd7\" (UniqueName: \"kubernetes.io/projected/159089a9-fca1-4bf3-ba81-feab46584505-kube-api-access-4pgd7\") pod \"payload-pre-processing-e2e-mt-f8439f5b-58b76f4b8c-x2dl8\" (UID: \"159089a9-fca1-4bf3-ba81-feab46584505\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f8439f5b-58b76f4b8c-x2dl8"
Nov 03 09:49:55.940811 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:55.940625    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/159089a9-fca1-4bf3-ba81-feab46584505-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f8439f5b-58b76f4b8c-x2dl8\" (UID: \"159089a9-fca1-4bf3-ba81-feab46584505\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f8439f5b-58b76f4b8c-x2dl8"
Nov 03 09:49:55.964449 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:55.964359    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f8439f5b-64dd678df-22vnq"]
Nov 03 09:49:55.969734 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:55.969710    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f8439f5b-64dd678df-22vnq"
Nov 03 09:49:55.995693 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:55.995666    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f8439f5b-64dd678df-22vnq"]
Nov 03 09:49:56.042169 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:56.042138    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80516551-d2d3-4bb6-897a-6da41d71e5c8-plugins-config-volume\") pod \"payload-processing-e2e-mt-f8439f5b-64dd678df-22vnq\" (UID: \"80516551-d2d3-4bb6-897a-6da41d71e5c8\") " pod="openshift-ingress/payload-processing-e2e-mt-f8439f5b-64dd678df-22vnq"
Nov 03 09:49:56.042325 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:56.042203    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pgd7\" (UniqueName: \"kubernetes.io/projected/159089a9-fca1-4bf3-ba81-feab46584505-kube-api-access-4pgd7\") pod \"payload-pre-processing-e2e-mt-f8439f5b-58b76f4b8c-x2dl8\" (UID: \"159089a9-fca1-4bf3-ba81-feab46584505\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f8439f5b-58b76f4b8c-x2dl8"
Nov 03 09:49:56.042325 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:56.042281    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/159089a9-fca1-4bf3-ba81-feab46584505-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f8439f5b-58b76f4b8c-x2dl8\" (UID: \"159089a9-fca1-4bf3-ba81-feab46584505\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f8439f5b-58b76f4b8c-x2dl8"
Nov 03 09:49:56.042410 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:56.042343    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd2pk\" (UniqueName: \"kubernetes.io/projected/80516551-d2d3-4bb6-897a-6da41d71e5c8-kube-api-access-gd2pk\") pod \"payload-processing-e2e-mt-f8439f5b-64dd678df-22vnq\" (UID: \"80516551-d2d3-4bb6-897a-6da41d71e5c8\") " pod="openshift-ingress/payload-processing-e2e-mt-f8439f5b-64dd678df-22vnq"
Nov 03 09:49:56.042992 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:56.042925    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/159089a9-fca1-4bf3-ba81-feab46584505-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f8439f5b-58b76f4b8c-x2dl8\" (UID: \"159089a9-fca1-4bf3-ba81-feab46584505\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f8439f5b-58b76f4b8c-x2dl8"
Nov 03 09:49:56.101563 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:56.101530    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pgd7\" (UniqueName: \"kubernetes.io/projected/159089a9-fca1-4bf3-ba81-feab46584505-kube-api-access-4pgd7\") pod \"payload-pre-processing-e2e-mt-f8439f5b-58b76f4b8c-x2dl8\" (UID: \"159089a9-fca1-4bf3-ba81-feab46584505\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f8439f5b-58b76f4b8c-x2dl8"
Nov 03 09:49:56.109653 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:56.109629    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8439f5b-58b76f4b8c-x2dl8"
Nov 03 09:49:56.143165 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:56.143138    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd2pk\" (UniqueName: \"kubernetes.io/projected/80516551-d2d3-4bb6-897a-6da41d71e5c8-kube-api-access-gd2pk\") pod \"payload-processing-e2e-mt-f8439f5b-64dd678df-22vnq\" (UID: \"80516551-d2d3-4bb6-897a-6da41d71e5c8\") " pod="openshift-ingress/payload-processing-e2e-mt-f8439f5b-64dd678df-22vnq"
Nov 03 09:49:56.143305 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:56.143254    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80516551-d2d3-4bb6-897a-6da41d71e5c8-plugins-config-volume\") pod \"payload-processing-e2e-mt-f8439f5b-64dd678df-22vnq\" (UID: \"80516551-d2d3-4bb6-897a-6da41d71e5c8\") " pod="openshift-ingress/payload-processing-e2e-mt-f8439f5b-64dd678df-22vnq"
Nov 03 09:49:56.143926 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:56.143904    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80516551-d2d3-4bb6-897a-6da41d71e5c8-plugins-config-volume\") pod \"payload-processing-e2e-mt-f8439f5b-64dd678df-22vnq\" (UID: \"80516551-d2d3-4bb6-897a-6da41d71e5c8\") " pod="openshift-ingress/payload-processing-e2e-mt-f8439f5b-64dd678df-22vnq"
Nov 03 09:49:56.202964 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:56.202913    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd2pk\" (UniqueName: \"kubernetes.io/projected/80516551-d2d3-4bb6-897a-6da41d71e5c8-kube-api-access-gd2pk\") pod \"payload-processing-e2e-mt-f8439f5b-64dd678df-22vnq\" (UID: \"80516551-d2d3-4bb6-897a-6da41d71e5c8\") " pod="openshift-ingress/payload-processing-e2e-mt-f8439f5b-64dd678df-22vnq"
Nov 03 09:49:56.292742 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:56.292709    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f8439f5b-64dd678df-22vnq"
Nov 03 09:49:56.488990 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:56.488962    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f8439f5b-58b76f4b8c-x2dl8"]
Nov 03 09:49:56.489753 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:49:56.489729    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod159089a9_fca1_4bf3_ba81_feab46584505.slice/crio-5d6cfd345a1cb123e266ab5fa1c8610f601483ef97d87b7e2553bae27ebe422f WatchSource:0}: Error finding container 5d6cfd345a1cb123e266ab5fa1c8610f601483ef97d87b7e2553bae27ebe422f: Status 404 returned error can't find the container with id 5d6cfd345a1cb123e266ab5fa1c8610f601483ef97d87b7e2553bae27ebe422f
Nov 03 09:49:56.668676 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:56.668653    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f8439f5b-64dd678df-22vnq"]
Nov 03 09:49:56.670085 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:49:56.670058    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80516551_d2d3_4bb6_897a_6da41d71e5c8.slice/crio-3a14bf1eea9ee031de6e7161e3ae02f433cb2888c6df35661f1f34efe58dcfed WatchSource:0}: Error finding container 3a14bf1eea9ee031de6e7161e3ae02f433cb2888c6df35661f1f34efe58dcfed: Status 404 returned error can't find the container with id 3a14bf1eea9ee031de6e7161e3ae02f433cb2888c6df35661f1f34efe58dcfed
Nov 03 09:49:57.058506 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:57.058467    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8439f5b-58b76f4b8c-x2dl8" event={"ID":"159089a9-fca1-4bf3-ba81-feab46584505","Type":"ContainerStarted","Data":"13de1eb7a83755a554dcb25aaec8d58854d09ce49300921e284d6d48588fb205"}
Nov 03 09:49:57.059021 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:57.058681    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8439f5b-58b76f4b8c-x2dl8"
Nov 03 09:49:57.059021 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:57.058723    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8439f5b-58b76f4b8c-x2dl8" event={"ID":"159089a9-fca1-4bf3-ba81-feab46584505","Type":"ContainerStarted","Data":"5d6cfd345a1cb123e266ab5fa1c8610f601483ef97d87b7e2553bae27ebe422f"}
Nov 03 09:49:57.062451 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:57.062430    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f8439f5b-64dd678df-22vnq" event={"ID":"80516551-d2d3-4bb6-897a-6da41d71e5c8","Type":"ContainerStarted","Data":"96311e5b248037420185a91ca922914253b660946276e8d1f0a1665e85af1ab3"}
Nov 03 09:49:57.062566 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:57.062465    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f8439f5b-64dd678df-22vnq" event={"ID":"80516551-d2d3-4bb6-897a-6da41d71e5c8","Type":"ContainerStarted","Data":"3a14bf1eea9ee031de6e7161e3ae02f433cb2888c6df35661f1f34efe58dcfed"}
Nov 03 09:49:57.062566 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:57.062525    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f8439f5b-64dd678df-22vnq"
Nov 03 09:49:57.175911 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:57.175857    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8439f5b-58b76f4b8c-x2dl8" podStartSLOduration=2.175843562 podStartE2EDuration="2.175843562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:49:57.174041462 +0000 UTC m=+4221.443104896" watchObservedRunningTime="2026-11-03 09:49:57.175843562 +0000 UTC m=+4221.444906996"
Nov 03 09:49:57.303227 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:57.303119    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f8439f5b-64dd678df-22vnq" podStartSLOduration=2.303099321 podStartE2EDuration="2.303099321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:49:57.301669183 +0000 UTC m=+4221.570732683" watchObservedRunningTime="2026-11-03 09:49:57.303099321 +0000 UTC m=+4221.572162756"
Nov 03 09:49:58.684114 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:49:58.684071    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3df1cc-6d58d67765-44gqp" podUID="9f2cb2de-df5b-4f79-95f1-4232463e82bb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:50:06.886662 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:06.886628    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3df1cc-6d58d67765-44gqp"
Nov 03 09:50:07.063978 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:07.063922    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f2cb2de-df5b-4f79-95f1-4232463e82bb-kube-api-access-zdf54\" (UniqueName: \"kubernetes.io/projected/9f2cb2de-df5b-4f79-95f1-4232463e82bb-kube-api-access-zdf54\") pod \"9f2cb2de-df5b-4f79-95f1-4232463e82bb\" (UID: \"9f2cb2de-df5b-4f79-95f1-4232463e82bb\") "
Nov 03 09:50:07.064170 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:07.064095    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f2cb2de-df5b-4f79-95f1-4232463e82bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f2cb2de-df5b-4f79-95f1-4232463e82bb-plugins-config-volume\") pod \"9f2cb2de-df5b-4f79-95f1-4232463e82bb\" (UID: \"9f2cb2de-df5b-4f79-95f1-4232463e82bb\") "
Nov 03 09:50:07.064455 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:07.064434    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f2cb2de-df5b-4f79-95f1-4232463e82bb-plugins-config-volume" pod "9f2cb2de-df5b-4f79-95f1-4232463e82bb" (UID: "9f2cb2de-df5b-4f79-95f1-4232463e82bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:50:07.066218 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:07.066196    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f2cb2de-df5b-4f79-95f1-4232463e82bb-kube-api-access-zdf54" pod "9f2cb2de-df5b-4f79-95f1-4232463e82bb" (UID: "9f2cb2de-df5b-4f79-95f1-4232463e82bb"). InnerVolumeSpecName "kube-api-access-zdf54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:50:07.123276 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:07.123195    4473 generic.go:356] "Generic (PLEG): container finished" podID="9f2cb2de-df5b-4f79-95f1-4232463e82bb" containerID="9b04779f0939659e793305130e31d8046b84cd548035b00bb776d99cb6770ec1" exitCode=0
Nov 03 09:50:07.123276 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:07.123240    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3df1cc-6d58d67765-44gqp"
Nov 03 09:50:07.123483 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:07.123280    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3df1cc-6d58d67765-44gqp" event={"ID":"9f2cb2de-df5b-4f79-95f1-4232463e82bb","Type":"ContainerDied","Data":"9b04779f0939659e793305130e31d8046b84cd548035b00bb776d99cb6770ec1"}
Nov 03 09:50:07.123483 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:07.123334    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3df1cc-6d58d67765-44gqp" event={"ID":"9f2cb2de-df5b-4f79-95f1-4232463e82bb","Type":"ContainerDied","Data":"b7eda0b52c99f21fd663ad57e37bdc264cbedcb5d30f956211774a7ad609688d"}
Nov 03 09:50:07.123483 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:07.123355    4473 scope.go:122] "RemoveContainer" containerID="9b04779f0939659e793305130e31d8046b84cd548035b00bb776d99cb6770ec1"
Nov 03 09:50:07.136479 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:07.136460    4473 scope.go:122] "RemoveContainer" containerID="9b04779f0939659e793305130e31d8046b84cd548035b00bb776d99cb6770ec1"
Nov 03 09:50:07.136783 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:50:07.136766    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b04779f0939659e793305130e31d8046b84cd548035b00bb776d99cb6770ec1\": container with ID starting with 9b04779f0939659e793305130e31d8046b84cd548035b00bb776d99cb6770ec1 not found: ID does not exist" containerID="9b04779f0939659e793305130e31d8046b84cd548035b00bb776d99cb6770ec1"
Nov 03 09:50:07.136831 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:07.136793    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b04779f0939659e793305130e31d8046b84cd548035b00bb776d99cb6770ec1"} err="failed to get container status \"9b04779f0939659e793305130e31d8046b84cd548035b00bb776d99cb6770ec1\": rpc error: code = NotFound desc = could not find container \"9b04779f0939659e793305130e31d8046b84cd548035b00bb776d99cb6770ec1\": container with ID starting with 9b04779f0939659e793305130e31d8046b84cd548035b00bb776d99cb6770ec1 not found: ID does not exist"
Nov 03 09:50:07.165826 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:07.165791    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdf54\" (UniqueName: \"kubernetes.io/projected/9f2cb2de-df5b-4f79-95f1-4232463e82bb-kube-api-access-zdf54\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:50:07.165826 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:07.165822    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f2cb2de-df5b-4f79-95f1-4232463e82bb-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:50:07.292788 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:07.292736    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-3df1cc-6d58d67765-44gqp"]
Nov 03 09:50:07.305152 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:07.305117    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-3df1cc-6d58d67765-44gqp"]
Nov 03 09:50:07.870277 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:07.870244    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f2cb2de-df5b-4f79-95f1-4232463e82bb" path="/var/lib/kubelet/pods/9f2cb2de-df5b-4f79-95f1-4232463e82bb/volumes"
Nov 03 09:50:08.071127 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:08.071097    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f8439f5b-64dd678df-22vnq"
Nov 03 09:50:08.071541 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:08.071161    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8439f5b-58b76f4b8c-x2dl8"
Nov 03 09:50:13.208355 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:13.208324    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:50:16.937690 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:50:16.937652    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:50:25.053157 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:25.053118    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f9981fd-t4djq"]
Nov 03 09:50:25.103297 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:25.103271    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f9981fd-t4djq"]
Nov 03 09:50:25.870450 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:25.870418    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b2df33a-75de-45f9-b7c4-45f13ab82990" path="/var/lib/kubelet/pods/1b2df33a-75de-45f9-b7c4-45f13ab82990/volumes"
Nov 03 09:50:38.857283 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:38.857255    4473 scope.go:122] "RemoveContainer" containerID="83e94e8b67d401656ea6f82e13a699591962148c564b4a1ebd07d00c685b485f"
Nov 03 09:50:38.867314 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:38.867290    4473 scope.go:122] "RemoveContainer" containerID="d058e599b9ee318faa84389936b48a805a2a9910c56b8a0f4fe27c697812a2bf"
Nov 03 09:50:44.990096 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:44.990066    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"]
Nov 03 09:50:44.991089 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:44.990316    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v" podUID="603f6f44-b6ae-4d49-b79d-e73bb6d853a9" containerName="istio-proxy" containerID="cri-o://c74561f0f1d70f02fae8c7509916bdb5b44ad74f364c8a964f63c3475459b4a1" gracePeriod=30
Nov 03 09:50:46.149821 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:46.149786    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18a7390b-h2tcj"]
Nov 03 09:50:46.150331 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:46.150317    4473 cpu_manager.go:414] "containerMap: removing container" podUID="9f2cb2de-df5b-4f79-95f1-4232463e82bb" containerName="payload-processing"
Nov 03 09:50:46.150381 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:46.150332    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f2cb2de-df5b-4f79-95f1-4232463e82bb" containerName="payload-processing"
Nov 03 09:50:46.150419 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:46.150413    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f2cb2de-df5b-4f79-95f1-4232463e82bb" containerName="payload-processing"
Nov 03 09:50:46.153523 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:46.153495    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a7390b-h2tcj"
Nov 03 09:50:46.211928 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:46.211899    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18a7390b-h2tcj"]
Nov 03 09:50:46.253146 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:46.253112    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92f3682c-4935-4a79-824a-6db5f2ac5c53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18a7390b-h2tcj\" (UID: \"92f3682c-4935-4a79-824a-6db5f2ac5c53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a7390b-h2tcj"
Nov 03 09:50:46.253324 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:46.253172    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92f3682c-4935-4a79-824a-6db5f2ac5c53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18a7390b-h2tcj\" (UID: \"92f3682c-4935-4a79-824a-6db5f2ac5c53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a7390b-h2tcj"
Nov 03 09:50:46.253324 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:46.253275    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntmwt\" (UniqueName: \"kubernetes.io/projected/92f3682c-4935-4a79-824a-6db5f2ac5c53-kube-api-access-ntmwt\") pod \"maas-api-revoke-sub-18a7390b-h2tcj\" (UID: \"92f3682c-4935-4a79-824a-6db5f2ac5c53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a7390b-h2tcj"
Nov 03 09:50:46.258247 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:46.258223    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:50:46.258328 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:46.258244    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:50:46.258471 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:46.258455    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:50:46.258514 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:46.258488    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:50:46.354539 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:46.354503    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92f3682c-4935-4a79-824a-6db5f2ac5c53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18a7390b-h2tcj\" (UID: \"92f3682c-4935-4a79-824a-6db5f2ac5c53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a7390b-h2tcj"
Nov 03 09:50:46.354724 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:46.354559    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92f3682c-4935-4a79-824a-6db5f2ac5c53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18a7390b-h2tcj\" (UID: \"92f3682c-4935-4a79-824a-6db5f2ac5c53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a7390b-h2tcj"
Nov 03 09:50:46.354724 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:46.354612    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntmwt\" (UniqueName: \"kubernetes.io/projected/92f3682c-4935-4a79-824a-6db5f2ac5c53-kube-api-access-ntmwt\") pod \"maas-api-revoke-sub-18a7390b-h2tcj\" (UID: \"92f3682c-4935-4a79-824a-6db5f2ac5c53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a7390b-h2tcj"
Nov 03 09:50:46.355381 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:46.355317    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92f3682c-4935-4a79-824a-6db5f2ac5c53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18a7390b-h2tcj\" (UID: \"92f3682c-4935-4a79-824a-6db5f2ac5c53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a7390b-h2tcj"
Nov 03 09:50:46.357163 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:46.357144    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92f3682c-4935-4a79-824a-6db5f2ac5c53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18a7390b-h2tcj\" (UID: \"92f3682c-4935-4a79-824a-6db5f2ac5c53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a7390b-h2tcj"
Nov 03 09:50:46.413702 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:46.413643    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntmwt\" (UniqueName: \"kubernetes.io/projected/92f3682c-4935-4a79-824a-6db5f2ac5c53-kube-api-access-ntmwt\") pod \"maas-api-revoke-sub-18a7390b-h2tcj\" (UID: \"92f3682c-4935-4a79-824a-6db5f2ac5c53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a7390b-h2tcj"
Nov 03 09:50:46.467850 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:46.467828    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a7390b-h2tcj"
Nov 03 09:50:46.843852 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:46.843822    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18a7390b-h2tcj"]
Nov 03 09:50:46.843852 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:50:46.843836    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92f3682c_4935_4a79_824a_6db5f2ac5c53.slice/crio-956e787f1ad6171a34968fcbcd4315745a3995963daa843d071a6f6a7e159c82 WatchSource:0}: Error finding container 956e787f1ad6171a34968fcbcd4315745a3995963daa843d071a6f6a7e159c82: Status 404 returned error can't find the container with id 956e787f1ad6171a34968fcbcd4315745a3995963daa843d071a6f6a7e159c82
Nov 03 09:50:46.937102 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:50:46.937031    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:50:47.382100 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:47.382017    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a7390b-h2tcj" event={"ID":"92f3682c-4935-4a79-824a-6db5f2ac5c53","Type":"ContainerStarted","Data":"8fe90596c830a0795702f1c73d74da57d29f5d4650912193f9d9765534f9d132"}
Nov 03 09:50:47.382100 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:47.382073    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a7390b-h2tcj" event={"ID":"92f3682c-4935-4a79-824a-6db5f2ac5c53","Type":"ContainerStarted","Data":"956e787f1ad6171a34968fcbcd4315745a3995963daa843d071a6f6a7e159c82"}
Nov 03 09:50:47.504282 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:47.504236    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a7390b-h2tcj" podStartSLOduration=1.504221129 podStartE2EDuration="1.504221129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:50:47.500924965 +0000 UTC m=+4271.769988415" watchObservedRunningTime="2026-11-03 09:50:47.504221129 +0000 UTC m=+4271.773284563"
Nov 03 09:50:48.389825 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:48.389789    4473 generic.go:356] "Generic (PLEG): container finished" podID="92f3682c-4935-4a79-824a-6db5f2ac5c53" containerID="8fe90596c830a0795702f1c73d74da57d29f5d4650912193f9d9765534f9d132" exitCode=0
Nov 03 09:50:48.389825 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:48.389826    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a7390b-h2tcj" event={"ID":"92f3682c-4935-4a79-824a-6db5f2ac5c53","Type":"ContainerDied","Data":"8fe90596c830a0795702f1c73d74da57d29f5d4650912193f9d9765534f9d132"}
Nov 03 09:50:49.548232 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:49.548209    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a7390b-h2tcj"
Nov 03 09:50:49.695404 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:49.695325    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92f3682c-4935-4a79-824a-6db5f2ac5c53-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92f3682c-4935-4a79-824a-6db5f2ac5c53-maas-api-openshift-ca\") pod \"92f3682c-4935-4a79-824a-6db5f2ac5c53\" (UID: \"92f3682c-4935-4a79-824a-6db5f2ac5c53\") "
Nov 03 09:50:49.695572 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:49.695410    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92f3682c-4935-4a79-824a-6db5f2ac5c53-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92f3682c-4935-4a79-824a-6db5f2ac5c53-maas-api-xks-ca\") pod \"92f3682c-4935-4a79-824a-6db5f2ac5c53\" (UID: \"92f3682c-4935-4a79-824a-6db5f2ac5c53\") "
Nov 03 09:50:49.695572 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:49.695475    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92f3682c-4935-4a79-824a-6db5f2ac5c53-kube-api-access-ntmwt\" (UniqueName: \"kubernetes.io/projected/92f3682c-4935-4a79-824a-6db5f2ac5c53-kube-api-access-ntmwt\") pod \"92f3682c-4935-4a79-824a-6db5f2ac5c53\" (UID: \"92f3682c-4935-4a79-824a-6db5f2ac5c53\") "
Nov 03 09:50:49.695751 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:49.695722    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92f3682c-4935-4a79-824a-6db5f2ac5c53-maas-api-openshift-ca" pod "92f3682c-4935-4a79-824a-6db5f2ac5c53" (UID: "92f3682c-4935-4a79-824a-6db5f2ac5c53"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:50:49.696107 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:49.696085    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92f3682c-4935-4a79-824a-6db5f2ac5c53-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:50:49.698755 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:49.698142    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92f3682c-4935-4a79-824a-6db5f2ac5c53-maas-api-xks-ca" pod "92f3682c-4935-4a79-824a-6db5f2ac5c53" (UID: "92f3682c-4935-4a79-824a-6db5f2ac5c53"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:50:49.698755 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:49.698444    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92f3682c-4935-4a79-824a-6db5f2ac5c53-kube-api-access-ntmwt" pod "92f3682c-4935-4a79-824a-6db5f2ac5c53" (UID: "92f3682c-4935-4a79-824a-6db5f2ac5c53"). InnerVolumeSpecName "kube-api-access-ntmwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:50:49.797091 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:49.797062    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92f3682c-4935-4a79-824a-6db5f2ac5c53-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:50:49.797091 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:49.797091    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntmwt\" (UniqueName: \"kubernetes.io/projected/92f3682c-4935-4a79-824a-6db5f2ac5c53-kube-api-access-ntmwt\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:50:50.243931 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.243904    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"
Nov 03 09:50:50.369628 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.369593    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn"]
Nov 03 09:50:50.403743 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.403721    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a7390b-h2tcj"
Nov 03 09:50:50.403931 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.403756    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a7390b-h2tcj" event={"ID":"92f3682c-4935-4a79-824a-6db5f2ac5c53","Type":"ContainerDied","Data":"956e787f1ad6171a34968fcbcd4315745a3995963daa843d071a6f6a7e159c82"}
Nov 03 09:50:50.403931 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.403805    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="956e787f1ad6171a34968fcbcd4315745a3995963daa843d071a6f6a7e159c82"
Nov 03 09:50:50.404118 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.404044    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-istiod-ca-cert\") pod \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\" (UID: \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\") "
Nov 03 09:50:50.404118 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.404086    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-workload-socket\") pod \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\" (UID: \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\") "
Nov 03 09:50:50.404234 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.404123    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-istio-podinfo\") pod \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\" (UID: \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\") "
Nov 03 09:50:50.404234 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.404205    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-istio-envoy\") pod \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\" (UID: \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\") "
Nov 03 09:50:50.404345 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.404249    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-workload-certs\") pod \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\" (UID: \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\") "
Nov 03 09:50:50.404345 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.404290    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-istio-token\" (UniqueName: \"kubernetes.io/projected/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-istio-token\") pod \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\" (UID: \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\") "
Nov 03 09:50:50.404345 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.404323    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-istio-data\") pod \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\" (UID: \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\") "
Nov 03 09:50:50.404486 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.404361    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-credential-socket\") pod \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\" (UID: \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\") "
Nov 03 09:50:50.404486 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.404435    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-istiod-ca-cert" pod "603f6f44-b6ae-4d49-b79d-e73bb6d853a9" (UID: "603f6f44-b6ae-4d49-b79d-e73bb6d853a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:50:50.404603 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.404548    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-kube-api-access-cf4p8\" (UniqueName: \"kubernetes.io/projected/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-kube-api-access-cf4p8\") pod \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\" (UID: \"603f6f44-b6ae-4d49-b79d-e73bb6d853a9\") "
Nov 03 09:50:50.405087 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.405014    4473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-istiod-ca-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:50:50.405473 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.405436    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-workload-socket" pod "603f6f44-b6ae-4d49-b79d-e73bb6d853a9" (UID: "603f6f44-b6ae-4d49-b79d-e73bb6d853a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:50:50.406000 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.405940    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-workload-certs" pod "603f6f44-b6ae-4d49-b79d-e73bb6d853a9" (UID: "603f6f44-b6ae-4d49-b79d-e73bb6d853a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:50:50.406225 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.406200    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-istio-data" pod "603f6f44-b6ae-4d49-b79d-e73bb6d853a9" (UID: "603f6f44-b6ae-4d49-b79d-e73bb6d853a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:50:50.406474 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.406448    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-credential-socket" pod "603f6f44-b6ae-4d49-b79d-e73bb6d853a9" (UID: "603f6f44-b6ae-4d49-b79d-e73bb6d853a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:50:50.408248 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.408199    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-istio-envoy" pod "603f6f44-b6ae-4d49-b79d-e73bb6d853a9" (UID: "603f6f44-b6ae-4d49-b79d-e73bb6d853a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:50:50.408355 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.408261    4473 generic.go:356] "Generic (PLEG): container finished" podID="603f6f44-b6ae-4d49-b79d-e73bb6d853a9" containerID="c74561f0f1d70f02fae8c7509916bdb5b44ad74f364c8a964f63c3475459b4a1" exitCode=0
Nov 03 09:50:50.408427 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.408398    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v" event={"ID":"603f6f44-b6ae-4d49-b79d-e73bb6d853a9","Type":"ContainerDied","Data":"c74561f0f1d70f02fae8c7509916bdb5b44ad74f364c8a964f63c3475459b4a1"}
Nov 03 09:50:50.408494 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.408440    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v" event={"ID":"603f6f44-b6ae-4d49-b79d-e73bb6d853a9","Type":"ContainerDied","Data":"46deb179715cbf91fc7f043e4ba26a6e0a52e0b5be0b95c664abd2833a5125d6"}
Nov 03 09:50:50.408494 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.408468    4473 scope.go:122] "RemoveContainer" containerID="c74561f0f1d70f02fae8c7509916bdb5b44ad74f364c8a964f63c3475459b4a1"
Nov 03 09:50:50.408632 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.408617    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"
Nov 03 09:50:50.409200 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.409046    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-kube-api-access-cf4p8" pod "603f6f44-b6ae-4d49-b79d-e73bb6d853a9" (UID: "603f6f44-b6ae-4d49-b79d-e73bb6d853a9"). InnerVolumeSpecName "kube-api-access-cf4p8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:50:50.409200 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.409104    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-istio-podinfo" pod "603f6f44-b6ae-4d49-b79d-e73bb6d853a9" (UID: "603f6f44-b6ae-4d49-b79d-e73bb6d853a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 09:50:50.409200 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.409136    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-istio-token" pod "603f6f44-b6ae-4d49-b79d-e73bb6d853a9" (UID: "603f6f44-b6ae-4d49-b79d-e73bb6d853a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:50:50.420914 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.420897    4473 scope.go:122] "RemoveContainer" containerID="6651d9c20d2ae8db25f249ae0b3c4ed341bc86f976668be7c11ec8b1183a53d1"
Nov 03 09:50:50.430449 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.430432    4473 scope.go:122] "RemoveContainer" containerID="c74561f0f1d70f02fae8c7509916bdb5b44ad74f364c8a964f63c3475459b4a1"
Nov 03 09:50:50.430764 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:50:50.430743    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c74561f0f1d70f02fae8c7509916bdb5b44ad74f364c8a964f63c3475459b4a1\": container with ID starting with c74561f0f1d70f02fae8c7509916bdb5b44ad74f364c8a964f63c3475459b4a1 not found: ID does not exist" containerID="c74561f0f1d70f02fae8c7509916bdb5b44ad74f364c8a964f63c3475459b4a1"
Nov 03 09:50:50.430814 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.430770    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c74561f0f1d70f02fae8c7509916bdb5b44ad74f364c8a964f63c3475459b4a1"} err="failed to get container status \"c74561f0f1d70f02fae8c7509916bdb5b44ad74f364c8a964f63c3475459b4a1\": rpc error: code = NotFound desc = could not find container \"c74561f0f1d70f02fae8c7509916bdb5b44ad74f364c8a964f63c3475459b4a1\": container with ID starting with c74561f0f1d70f02fae8c7509916bdb5b44ad74f364c8a964f63c3475459b4a1 not found: ID does not exist"
Nov 03 09:50:50.430814 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.430787    4473 scope.go:122] "RemoveContainer" containerID="6651d9c20d2ae8db25f249ae0b3c4ed341bc86f976668be7c11ec8b1183a53d1"
Nov 03 09:50:50.431068 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:50:50.431050    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6651d9c20d2ae8db25f249ae0b3c4ed341bc86f976668be7c11ec8b1183a53d1\": container with ID starting with 6651d9c20d2ae8db25f249ae0b3c4ed341bc86f976668be7c11ec8b1183a53d1 not found: ID does not exist" containerID="6651d9c20d2ae8db25f249ae0b3c4ed341bc86f976668be7c11ec8b1183a53d1"
Nov 03 09:50:50.431124 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.431076    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6651d9c20d2ae8db25f249ae0b3c4ed341bc86f976668be7c11ec8b1183a53d1"} err="failed to get container status \"6651d9c20d2ae8db25f249ae0b3c4ed341bc86f976668be7c11ec8b1183a53d1\": rpc error: code = NotFound desc = could not find container \"6651d9c20d2ae8db25f249ae0b3c4ed341bc86f976668be7c11ec8b1183a53d1\": container with ID starting with 6651d9c20d2ae8db25f249ae0b3c4ed341bc86f976668be7c11ec8b1183a53d1 not found: ID does not exist"
Nov 03 09:50:50.506528 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.506502    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf4p8\" (UniqueName: \"kubernetes.io/projected/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-kube-api-access-cf4p8\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:50:50.506528 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.506526    4473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-workload-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:50:50.506718 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.506537    4473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-istio-podinfo\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:50:50.506718 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.506546    4473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-istio-envoy\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:50:50.506718 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.506554    4473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-workload-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:50:50.506718 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.506565    4473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-istio-token\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:50:50.506718 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.506573    4473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-istio-data\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:50:50.506718 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.506581    4473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/603f6f44-b6ae-4d49-b79d-e73bb6d853a9-credential-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:50:50.833162 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.833130    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"]
Nov 03 09:50:50.839963 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:50.839917    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3df1cc-openshift-default-77f9b4f49f-9rk4v"]
Nov 03 09:50:51.871469 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:50:51.871439    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="603f6f44-b6ae-4d49-b79d-e73bb6d853a9" path="/var/lib/kubelet/pods/603f6f44-b6ae-4d49-b79d-e73bb6d853a9/volumes"
Nov 03 09:51:02.520345 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:02.520314    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:51:02.553111 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:02.553063    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dee3e12-9twtp"]
Nov 03 09:51:02.553911 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:02.553893    4473 cpu_manager.go:414] "containerMap: removing container" podUID="603f6f44-b6ae-4d49-b79d-e73bb6d853a9" containerName="istio-proxy"
Nov 03 09:51:02.553980 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:02.553916    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="603f6f44-b6ae-4d49-b79d-e73bb6d853a9" containerName="istio-proxy"
Nov 03 09:51:02.553980 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:02.553932    4473 cpu_manager.go:414] "containerMap: removing container" podUID="603f6f44-b6ae-4d49-b79d-e73bb6d853a9" containerName="istio-proxy"
Nov 03 09:51:02.553980 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:02.553958    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="603f6f44-b6ae-4d49-b79d-e73bb6d853a9" containerName="istio-proxy"
Nov 03 09:51:02.553980 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:02.553972    4473 cpu_manager.go:414] "containerMap: removing container" podUID="92f3682c-4935-4a79-824a-6db5f2ac5c53" containerName="revoke-keys"
Nov 03 09:51:02.554111 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:02.553980    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92f3682c-4935-4a79-824a-6db5f2ac5c53" containerName="revoke-keys"
Nov 03 09:51:02.554149 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:02.554117    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="603f6f44-b6ae-4d49-b79d-e73bb6d853a9" containerName="istio-proxy"
Nov 03 09:51:02.554149 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:02.554130    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="92f3682c-4935-4a79-824a-6db5f2ac5c53" containerName="revoke-keys"
Nov 03 09:51:02.559318 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:02.559301    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dee3e12-9twtp"
Nov 03 09:51:02.614104 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:02.614060    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:51:02.614246 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:02.614160    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:51:02.614246 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:02.614164    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:51:02.614246 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:02.614160    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:51:02.631934 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:02.631909    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dee3e12-9twtp"]
Nov 03 09:51:02.651119 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:02.651096    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0de59ed1-428a-452f-a5ae-eacb005a59d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dee3e12-9twtp\" (UID: \"0de59ed1-428a-452f-a5ae-eacb005a59d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dee3e12-9twtp"
Nov 03 09:51:02.651246 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:02.651150    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnqrz\" (UniqueName: \"kubernetes.io/projected/0de59ed1-428a-452f-a5ae-eacb005a59d2-kube-api-access-bnqrz\") pod \"maas-api-revoke-sub-9dee3e12-9twtp\" (UID: \"0de59ed1-428a-452f-a5ae-eacb005a59d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dee3e12-9twtp"
Nov 03 09:51:02.651350 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:02.651275    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0de59ed1-428a-452f-a5ae-eacb005a59d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dee3e12-9twtp\" (UID: \"0de59ed1-428a-452f-a5ae-eacb005a59d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dee3e12-9twtp"
Nov 03 09:51:02.751883 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:02.751855    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0de59ed1-428a-452f-a5ae-eacb005a59d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dee3e12-9twtp\" (UID: \"0de59ed1-428a-452f-a5ae-eacb005a59d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dee3e12-9twtp"
Nov 03 09:51:02.752057 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:02.751901    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnqrz\" (UniqueName: \"kubernetes.io/projected/0de59ed1-428a-452f-a5ae-eacb005a59d2-kube-api-access-bnqrz\") pod \"maas-api-revoke-sub-9dee3e12-9twtp\" (UID: \"0de59ed1-428a-452f-a5ae-eacb005a59d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dee3e12-9twtp"
Nov 03 09:51:02.752057 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:02.751922    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0de59ed1-428a-452f-a5ae-eacb005a59d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dee3e12-9twtp\" (UID: \"0de59ed1-428a-452f-a5ae-eacb005a59d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dee3e12-9twtp"
Nov 03 09:51:02.752649 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:02.752629    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0de59ed1-428a-452f-a5ae-eacb005a59d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dee3e12-9twtp\" (UID: \"0de59ed1-428a-452f-a5ae-eacb005a59d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dee3e12-9twtp"
Nov 03 09:51:02.754589 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:02.754569    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0de59ed1-428a-452f-a5ae-eacb005a59d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dee3e12-9twtp\" (UID: \"0de59ed1-428a-452f-a5ae-eacb005a59d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dee3e12-9twtp"
Nov 03 09:51:02.811108 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:02.811045    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnqrz\" (UniqueName: \"kubernetes.io/projected/0de59ed1-428a-452f-a5ae-eacb005a59d2-kube-api-access-bnqrz\") pod \"maas-api-revoke-sub-9dee3e12-9twtp\" (UID: \"0de59ed1-428a-452f-a5ae-eacb005a59d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dee3e12-9twtp"
Nov 03 09:51:02.878397 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:02.878366    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dee3e12-9twtp"
Nov 03 09:51:03.254730 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:03.254705    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dee3e12-9twtp"]
Nov 03 09:51:03.254876 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:51:03.254809    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0de59ed1_428a_452f_a5ae_eacb005a59d2.slice/crio-cf158db0ebffd400f279bbd9163b7bb69e2fc995fe1918aa63d9c70964bfceea WatchSource:0}: Error finding container cf158db0ebffd400f279bbd9163b7bb69e2fc995fe1918aa63d9c70964bfceea: Status 404 returned error can't find the container with id cf158db0ebffd400f279bbd9163b7bb69e2fc995fe1918aa63d9c70964bfceea
Nov 03 09:51:03.502376 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:03.502300    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dee3e12-9twtp" event={"ID":"0de59ed1-428a-452f-a5ae-eacb005a59d2","Type":"ContainerStarted","Data":"cf158db0ebffd400f279bbd9163b7bb69e2fc995fe1918aa63d9c70964bfceea"}
Nov 03 09:51:04.509892 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:04.509860    4473 generic.go:356] "Generic (PLEG): container finished" podID="0de59ed1-428a-452f-a5ae-eacb005a59d2" containerID="1c3b07db48115e915e7c243dfb2c563bc0c7354e064ac32029490bfe731f4b17" exitCode=0
Nov 03 09:51:04.510327 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:04.509971    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dee3e12-9twtp" event={"ID":"0de59ed1-428a-452f-a5ae-eacb005a59d2","Type":"ContainerDied","Data":"1c3b07db48115e915e7c243dfb2c563bc0c7354e064ac32029490bfe731f4b17"}
Nov 03 09:51:05.405120 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:05.401186    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn" podUID="05a8a676-3011-4597-9b71-b70c3387e2b9" containerName="main" containerID="cri-o://6490c4d695eff37244ab1ae606833c5f7b7630aa47932063b70f96f73129ea42" gracePeriod=45
Nov 03 09:51:05.664790 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:05.664735    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dee3e12-9twtp"
Nov 03 09:51:05.693577 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:05.693548    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0de59ed1-428a-452f-a5ae-eacb005a59d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0de59ed1-428a-452f-a5ae-eacb005a59d2-maas-api-xks-ca\") pod \"0de59ed1-428a-452f-a5ae-eacb005a59d2\" (UID: \"0de59ed1-428a-452f-a5ae-eacb005a59d2\") "
Nov 03 09:51:05.693795 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:05.693773    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0de59ed1-428a-452f-a5ae-eacb005a59d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0de59ed1-428a-452f-a5ae-eacb005a59d2-maas-api-openshift-ca\") pod \"0de59ed1-428a-452f-a5ae-eacb005a59d2\" (UID: \"0de59ed1-428a-452f-a5ae-eacb005a59d2\") "
Nov 03 09:51:05.694046 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:05.694027    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0de59ed1-428a-452f-a5ae-eacb005a59d2-kube-api-access-bnqrz\" (UniqueName: \"kubernetes.io/projected/0de59ed1-428a-452f-a5ae-eacb005a59d2-kube-api-access-bnqrz\") pod \"0de59ed1-428a-452f-a5ae-eacb005a59d2\" (UID: \"0de59ed1-428a-452f-a5ae-eacb005a59d2\") "
Nov 03 09:51:05.694211 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:05.694178    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0de59ed1-428a-452f-a5ae-eacb005a59d2-maas-api-openshift-ca" pod "0de59ed1-428a-452f-a5ae-eacb005a59d2" (UID: "0de59ed1-428a-452f-a5ae-eacb005a59d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:51:05.694644 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:05.694600    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0de59ed1-428a-452f-a5ae-eacb005a59d2-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:51:05.696076 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:05.696052    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0de59ed1-428a-452f-a5ae-eacb005a59d2-maas-api-xks-ca" pod "0de59ed1-428a-452f-a5ae-eacb005a59d2" (UID: "0de59ed1-428a-452f-a5ae-eacb005a59d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:51:05.696418 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:05.696396    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0de59ed1-428a-452f-a5ae-eacb005a59d2-kube-api-access-bnqrz" pod "0de59ed1-428a-452f-a5ae-eacb005a59d2" (UID: "0de59ed1-428a-452f-a5ae-eacb005a59d2"). InnerVolumeSpecName "kube-api-access-bnqrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:51:05.712202 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:05.712183    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn"
Nov 03 09:51:05.796155 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:05.795736    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05a8a676-3011-4597-9b71-b70c3387e2b9-tls-certs\" (UniqueName: \"kubernetes.io/secret/05a8a676-3011-4597-9b71-b70c3387e2b9-tls-certs\") pod \"05a8a676-3011-4597-9b71-b70c3387e2b9\" (UID: \"05a8a676-3011-4597-9b71-b70c3387e2b9\") "
Nov 03 09:51:05.796155 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:05.795792    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05a8a676-3011-4597-9b71-b70c3387e2b9-kube-api-access-nbxzp\" (UniqueName: \"kubernetes.io/projected/05a8a676-3011-4597-9b71-b70c3387e2b9-kube-api-access-nbxzp\") pod \"05a8a676-3011-4597-9b71-b70c3387e2b9\" (UID: \"05a8a676-3011-4597-9b71-b70c3387e2b9\") "
Nov 03 09:51:05.796155 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:05.795889    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a8a676-3011-4597-9b71-b70c3387e2b9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05a8a676-3011-4597-9b71-b70c3387e2b9-model-cache\") pod \"05a8a676-3011-4597-9b71-b70c3387e2b9\" (UID: \"05a8a676-3011-4597-9b71-b70c3387e2b9\") "
Nov 03 09:51:05.796155 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:05.795965    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a8a676-3011-4597-9b71-b70c3387e2b9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05a8a676-3011-4597-9b71-b70c3387e2b9-tmp-dir\") pod \"05a8a676-3011-4597-9b71-b70c3387e2b9\" (UID: \"05a8a676-3011-4597-9b71-b70c3387e2b9\") "
Nov 03 09:51:05.796155 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:05.796025    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a8a676-3011-4597-9b71-b70c3387e2b9-home\" (UniqueName: \"kubernetes.io/empty-dir/05a8a676-3011-4597-9b71-b70c3387e2b9-home\") pod \"05a8a676-3011-4597-9b71-b70c3387e2b9\" (UID: \"05a8a676-3011-4597-9b71-b70c3387e2b9\") "
Nov 03 09:51:05.796155 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:05.796112    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a8a676-3011-4597-9b71-b70c3387e2b9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/05a8a676-3011-4597-9b71-b70c3387e2b9-dshm\") pod \"05a8a676-3011-4597-9b71-b70c3387e2b9\" (UID: \"05a8a676-3011-4597-9b71-b70c3387e2b9\") "
Nov 03 09:51:05.796687 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:05.796638    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnqrz\" (UniqueName: \"kubernetes.io/projected/0de59ed1-428a-452f-a5ae-eacb005a59d2-kube-api-access-bnqrz\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:51:05.796687 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:05.796663    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0de59ed1-428a-452f-a5ae-eacb005a59d2-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:51:05.796814 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:05.796669    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a8a676-3011-4597-9b71-b70c3387e2b9-model-cache" pod "05a8a676-3011-4597-9b71-b70c3387e2b9" (UID: "05a8a676-3011-4597-9b71-b70c3387e2b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:51:05.796814 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:05.796748    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a8a676-3011-4597-9b71-b70c3387e2b9-home" pod "05a8a676-3011-4597-9b71-b70c3387e2b9" (UID: "05a8a676-3011-4597-9b71-b70c3387e2b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:51:05.797026 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:05.797004    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a8a676-3011-4597-9b71-b70c3387e2b9-tmp-dir" pod "05a8a676-3011-4597-9b71-b70c3387e2b9" (UID: "05a8a676-3011-4597-9b71-b70c3387e2b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:51:05.799465 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:05.799439    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05a8a676-3011-4597-9b71-b70c3387e2b9-tls-certs" pod "05a8a676-3011-4597-9b71-b70c3387e2b9" (UID: "05a8a676-3011-4597-9b71-b70c3387e2b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:51:05.799591 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:05.799521    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a8a676-3011-4597-9b71-b70c3387e2b9-dshm" pod "05a8a676-3011-4597-9b71-b70c3387e2b9" (UID: "05a8a676-3011-4597-9b71-b70c3387e2b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:51:05.799591 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:05.799537    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05a8a676-3011-4597-9b71-b70c3387e2b9-kube-api-access-nbxzp" pod "05a8a676-3011-4597-9b71-b70c3387e2b9" (UID: "05a8a676-3011-4597-9b71-b70c3387e2b9"). InnerVolumeSpecName "kube-api-access-nbxzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:51:05.897689 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:05.897663    4473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05a8a676-3011-4597-9b71-b70c3387e2b9-tmp-dir\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:51:05.897689 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:05.897690    4473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05a8a676-3011-4597-9b71-b70c3387e2b9-home\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:51:05.897869 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:05.897698    4473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05a8a676-3011-4597-9b71-b70c3387e2b9-dshm\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:51:05.897869 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:05.897706    4473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05a8a676-3011-4597-9b71-b70c3387e2b9-tls-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:51:05.897869 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:05.897715    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbxzp\" (UniqueName: \"kubernetes.io/projected/05a8a676-3011-4597-9b71-b70c3387e2b9-kube-api-access-nbxzp\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:51:05.897869 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:05.897733    4473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05a8a676-3011-4597-9b71-b70c3387e2b9-model-cache\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:51:06.523549 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:06.523515    4473 generic.go:356] "Generic (PLEG): container finished" podID="05a8a676-3011-4597-9b71-b70c3387e2b9" containerID="6490c4d695eff37244ab1ae606833c5f7b7630aa47932063b70f96f73129ea42" exitCode=0
Nov 03 09:51:06.523728 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:06.523575    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn"
Nov 03 09:51:06.523728 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:06.523601    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn" event={"ID":"05a8a676-3011-4597-9b71-b70c3387e2b9","Type":"ContainerDied","Data":"6490c4d695eff37244ab1ae606833c5f7b7630aa47932063b70f96f73129ea42"}
Nov 03 09:51:06.523728 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:06.523654    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn" event={"ID":"05a8a676-3011-4597-9b71-b70c3387e2b9","Type":"ContainerDied","Data":"4656d7502ceea8c6c8903036d9a0a7cb164574383eb5cb1e3d1969953ee0c95b"}
Nov 03 09:51:06.523728 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:06.523676    4473 scope.go:122] "RemoveContainer" containerID="6490c4d695eff37244ab1ae606833c5f7b7630aa47932063b70f96f73129ea42"
Nov 03 09:51:06.525634 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:06.525597    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dee3e12-9twtp" event={"ID":"0de59ed1-428a-452f-a5ae-eacb005a59d2","Type":"ContainerDied","Data":"cf158db0ebffd400f279bbd9163b7bb69e2fc995fe1918aa63d9c70964bfceea"}
Nov 03 09:51:06.525743 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:06.525664    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf158db0ebffd400f279bbd9163b7bb69e2fc995fe1918aa63d9c70964bfceea"
Nov 03 09:51:06.525743 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:06.525710    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dee3e12-9twtp"
Nov 03 09:51:06.533797 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:06.533777    4473 scope.go:122] "RemoveContainer" containerID="6490c4d695eff37244ab1ae606833c5f7b7630aa47932063b70f96f73129ea42"
Nov 03 09:51:06.534103 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:51:06.534084    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6490c4d695eff37244ab1ae606833c5f7b7630aa47932063b70f96f73129ea42\": container with ID starting with 6490c4d695eff37244ab1ae606833c5f7b7630aa47932063b70f96f73129ea42 not found: ID does not exist" containerID="6490c4d695eff37244ab1ae606833c5f7b7630aa47932063b70f96f73129ea42"
Nov 03 09:51:06.534173 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:06.534112    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6490c4d695eff37244ab1ae606833c5f7b7630aa47932063b70f96f73129ea42"} err="failed to get container status \"6490c4d695eff37244ab1ae606833c5f7b7630aa47932063b70f96f73129ea42\": rpc error: code = NotFound desc = could not find container \"6490c4d695eff37244ab1ae606833c5f7b7630aa47932063b70f96f73129ea42\": container with ID starting with 6490c4d695eff37244ab1ae606833c5f7b7630aa47932063b70f96f73129ea42 not found: ID does not exist"
Nov 03 09:51:06.749095 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:06.749021    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn"]
Nov 03 09:51:06.798875 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:06.798848    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-26e861/distinct-2-w1-26e861-kserve-54ddb996c-2rxxn"]
Nov 03 09:51:07.870332 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:07.870296    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05a8a676-3011-4597-9b71-b70c3387e2b9" path="/var/lib/kubelet/pods/05a8a676-3011-4597-9b71-b70c3387e2b9/volumes"
Nov 03 09:51:16.937013 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:51:16.936975    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:51:30.761081 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:30.761049    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8ee58e4-vtsbx"]
Nov 03 09:51:30.761740 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:30.761718    4473 cpu_manager.go:414] "containerMap: removing container" podUID="05a8a676-3011-4597-9b71-b70c3387e2b9" containerName="main"
Nov 03 09:51:30.761838 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:30.761742    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05a8a676-3011-4597-9b71-b70c3387e2b9" containerName="main"
Nov 03 09:51:30.761838 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:30.761776    4473 cpu_manager.go:414] "containerMap: removing container" podUID="0de59ed1-428a-452f-a5ae-eacb005a59d2" containerName="revoke-keys"
Nov 03 09:51:30.761838 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:30.761784    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0de59ed1-428a-452f-a5ae-eacb005a59d2" containerName="revoke-keys"
Nov 03 09:51:30.762033 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:30.761938    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="05a8a676-3011-4597-9b71-b70c3387e2b9" containerName="main"
Nov 03 09:51:30.762033 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:30.761977    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="0de59ed1-428a-452f-a5ae-eacb005a59d2" containerName="revoke-keys"
Nov 03 09:51:30.762033 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:30.761994    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="603f6f44-b6ae-4d49-b79d-e73bb6d853a9" containerName="istio-proxy"
Nov 03 09:51:30.765288 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:30.765266    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ee58e4-vtsbx"
Nov 03 09:51:30.825965 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:30.825911    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8ee58e4-vtsbx"]
Nov 03 09:51:30.868351 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:30.868313    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:51:30.868351 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:30.868342    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:51:30.868582 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:30.868343    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:51:30.869094 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:30.869070    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:51:30.914000 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:30.913957    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2611f42a-9412-437a-8b84-fa09c47a91e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8ee58e4-vtsbx\" (UID: \"2611f42a-9412-437a-8b84-fa09c47a91e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ee58e4-vtsbx"
Nov 03 09:51:30.914154 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:30.914078    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2611f42a-9412-437a-8b84-fa09c47a91e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8ee58e4-vtsbx\" (UID: \"2611f42a-9412-437a-8b84-fa09c47a91e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ee58e4-vtsbx"
Nov 03 09:51:30.914197 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:30.914176    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgzvz\" (UniqueName: \"kubernetes.io/projected/2611f42a-9412-437a-8b84-fa09c47a91e6-kube-api-access-fgzvz\") pod \"maas-api-revoke-sub-f8ee58e4-vtsbx\" (UID: \"2611f42a-9412-437a-8b84-fa09c47a91e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ee58e4-vtsbx"
Nov 03 09:51:31.015782 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:31.015705    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2611f42a-9412-437a-8b84-fa09c47a91e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8ee58e4-vtsbx\" (UID: \"2611f42a-9412-437a-8b84-fa09c47a91e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ee58e4-vtsbx"
Nov 03 09:51:31.015782 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:31.015779    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgzvz\" (UniqueName: \"kubernetes.io/projected/2611f42a-9412-437a-8b84-fa09c47a91e6-kube-api-access-fgzvz\") pod \"maas-api-revoke-sub-f8ee58e4-vtsbx\" (UID: \"2611f42a-9412-437a-8b84-fa09c47a91e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ee58e4-vtsbx"
Nov 03 09:51:31.016039 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:31.015889    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2611f42a-9412-437a-8b84-fa09c47a91e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8ee58e4-vtsbx\" (UID: \"2611f42a-9412-437a-8b84-fa09c47a91e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ee58e4-vtsbx"
Nov 03 09:51:31.016567 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:31.016541    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2611f42a-9412-437a-8b84-fa09c47a91e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8ee58e4-vtsbx\" (UID: \"2611f42a-9412-437a-8b84-fa09c47a91e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ee58e4-vtsbx"
Nov 03 09:51:31.018529 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:31.018505    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2611f42a-9412-437a-8b84-fa09c47a91e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8ee58e4-vtsbx\" (UID: \"2611f42a-9412-437a-8b84-fa09c47a91e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ee58e4-vtsbx"
Nov 03 09:51:31.077505 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:31.077474    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgzvz\" (UniqueName: \"kubernetes.io/projected/2611f42a-9412-437a-8b84-fa09c47a91e6-kube-api-access-fgzvz\") pod \"maas-api-revoke-sub-f8ee58e4-vtsbx\" (UID: \"2611f42a-9412-437a-8b84-fa09c47a91e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ee58e4-vtsbx"
Nov 03 09:51:31.077677 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:31.077660    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ee58e4-vtsbx"
Nov 03 09:51:31.462609 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:31.462580    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8ee58e4-vtsbx"]
Nov 03 09:51:31.463119 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:51:31.463097    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2611f42a_9412_437a_8b84_fa09c47a91e6.slice/crio-48d9311418e4e804667e9693742adf882f446ed2d33e85a509bab1a247897805 WatchSource:0}: Error finding container 48d9311418e4e804667e9693742adf882f446ed2d33e85a509bab1a247897805: Status 404 returned error can't find the container with id 48d9311418e4e804667e9693742adf882f446ed2d33e85a509bab1a247897805
Nov 03 09:51:31.676749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:31.676712    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ee58e4-vtsbx" event={"ID":"2611f42a-9412-437a-8b84-fa09c47a91e6","Type":"ContainerStarted","Data":"48d9311418e4e804667e9693742adf882f446ed2d33e85a509bab1a247897805"}
Nov 03 09:51:32.684729 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:32.684694    4473 generic.go:356] "Generic (PLEG): container finished" podID="2611f42a-9412-437a-8b84-fa09c47a91e6" containerID="931563f03ed88d4dbf35983cdafd7ef70b9ae6e364189e3de30d52d8a7de1d25" exitCode=0
Nov 03 09:51:32.685215 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:32.684778    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ee58e4-vtsbx" event={"ID":"2611f42a-9412-437a-8b84-fa09c47a91e6","Type":"ContainerDied","Data":"931563f03ed88d4dbf35983cdafd7ef70b9ae6e364189e3de30d52d8a7de1d25"}
Nov 03 09:51:33.835601 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:33.835572    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ee58e4-vtsbx"
Nov 03 09:51:33.849388 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:33.849361    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2611f42a-9412-437a-8b84-fa09c47a91e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2611f42a-9412-437a-8b84-fa09c47a91e6-maas-api-openshift-ca\") pod \"2611f42a-9412-437a-8b84-fa09c47a91e6\" (UID: \"2611f42a-9412-437a-8b84-fa09c47a91e6\") "
Nov 03 09:51:33.849528 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:33.849409    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2611f42a-9412-437a-8b84-fa09c47a91e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2611f42a-9412-437a-8b84-fa09c47a91e6-maas-api-xks-ca\") pod \"2611f42a-9412-437a-8b84-fa09c47a91e6\" (UID: \"2611f42a-9412-437a-8b84-fa09c47a91e6\") "
Nov 03 09:51:33.849528 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:33.849439    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2611f42a-9412-437a-8b84-fa09c47a91e6-kube-api-access-fgzvz\" (UniqueName: \"kubernetes.io/projected/2611f42a-9412-437a-8b84-fa09c47a91e6-kube-api-access-fgzvz\") pod \"2611f42a-9412-437a-8b84-fa09c47a91e6\" (UID: \"2611f42a-9412-437a-8b84-fa09c47a91e6\") "
Nov 03 09:51:33.849724 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:33.849700    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2611f42a-9412-437a-8b84-fa09c47a91e6-maas-api-openshift-ca" pod "2611f42a-9412-437a-8b84-fa09c47a91e6" (UID: "2611f42a-9412-437a-8b84-fa09c47a91e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:51:33.849887 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:33.849870    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2611f42a-9412-437a-8b84-fa09c47a91e6-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:51:33.851577 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:33.851553    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2611f42a-9412-437a-8b84-fa09c47a91e6-maas-api-xks-ca" pod "2611f42a-9412-437a-8b84-fa09c47a91e6" (UID: "2611f42a-9412-437a-8b84-fa09c47a91e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:51:33.851666 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:33.851641    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2611f42a-9412-437a-8b84-fa09c47a91e6-kube-api-access-fgzvz" pod "2611f42a-9412-437a-8b84-fa09c47a91e6" (UID: "2611f42a-9412-437a-8b84-fa09c47a91e6"). InnerVolumeSpecName "kube-api-access-fgzvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:51:33.951369 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:33.951290    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2611f42a-9412-437a-8b84-fa09c47a91e6-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:51:33.951369 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:33.951327    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgzvz\" (UniqueName: \"kubernetes.io/projected/2611f42a-9412-437a-8b84-fa09c47a91e6-kube-api-access-fgzvz\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:51:34.697311 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:34.697277    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ee58e4-vtsbx"
Nov 03 09:51:34.697311 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:34.697299    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ee58e4-vtsbx" event={"ID":"2611f42a-9412-437a-8b84-fa09c47a91e6","Type":"ContainerDied","Data":"48d9311418e4e804667e9693742adf882f446ed2d33e85a509bab1a247897805"}
Nov 03 09:51:34.697525 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:34.697372    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48d9311418e4e804667e9693742adf882f446ed2d33e85a509bab1a247897805"
Nov 03 09:51:38.900722 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:38.900689    4473 scope.go:122] "RemoveContainer" containerID="db9f301444a0e60625c2e42f62c2d2be17e5eeca8d9e81a6805325d48cf20461"
Nov 03 09:51:41.506332 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:41.506301    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c77af362-hqs7b"]
Nov 03 09:51:41.506805 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:41.506791    4473 cpu_manager.go:414] "containerMap: removing container" podUID="2611f42a-9412-437a-8b84-fa09c47a91e6" containerName="revoke-keys"
Nov 03 09:51:41.506857 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:41.506806    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2611f42a-9412-437a-8b84-fa09c47a91e6" containerName="revoke-keys"
Nov 03 09:51:41.506916 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:41.506906    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="2611f42a-9412-437a-8b84-fa09c47a91e6" containerName="revoke-keys"
Nov 03 09:51:41.511370 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:41.511353    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c77af362-hqs7b"
Nov 03 09:51:41.567218 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:41.567185    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c77af362-hqs7b"]
Nov 03 09:51:41.614180 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:41.614155    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:51:41.614180 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:41.614155    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:51:41.614386 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:41.614195    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:51:41.615099 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:41.615087    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:51:41.651018 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:41.650988    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9qh2\" (UniqueName: \"kubernetes.io/projected/6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7-kube-api-access-p9qh2\") pod \"maas-api-revoke-sub-c77af362-hqs7b\" (UID: \"6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c77af362-hqs7b"
Nov 03 09:51:41.651176 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:41.651043    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c77af362-hqs7b\" (UID: \"6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c77af362-hqs7b"
Nov 03 09:51:41.651275 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:41.651254    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c77af362-hqs7b\" (UID: \"6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c77af362-hqs7b"
Nov 03 09:51:41.752208 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:41.752175    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c77af362-hqs7b\" (UID: \"6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c77af362-hqs7b"
Nov 03 09:51:41.752376 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:41.752233    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9qh2\" (UniqueName: \"kubernetes.io/projected/6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7-kube-api-access-p9qh2\") pod \"maas-api-revoke-sub-c77af362-hqs7b\" (UID: \"6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c77af362-hqs7b"
Nov 03 09:51:41.752376 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:41.752278    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c77af362-hqs7b\" (UID: \"6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c77af362-hqs7b"
Nov 03 09:51:41.753145 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:41.752976    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c77af362-hqs7b\" (UID: \"6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c77af362-hqs7b"
Nov 03 09:51:41.754773 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:41.754749    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c77af362-hqs7b\" (UID: \"6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c77af362-hqs7b"
Nov 03 09:51:41.811354 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:41.811294    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9qh2\" (UniqueName: \"kubernetes.io/projected/6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7-kube-api-access-p9qh2\") pod \"maas-api-revoke-sub-c77af362-hqs7b\" (UID: \"6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c77af362-hqs7b"
Nov 03 09:51:41.824262 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:41.824230    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c77af362-hqs7b"
Nov 03 09:51:42.203193 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:42.203162    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c77af362-hqs7b"]
Nov 03 09:51:42.203778 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:51:42.203756    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b50ede6_3ef0_400f_a3ca_c014a8e0f7b7.slice/crio-8c8c6c3358f9a3fededcca4d6e12c2ba7d6fa4d2e079c94d154c7b6d8f95bd6f WatchSource:0}: Error finding container 8c8c6c3358f9a3fededcca4d6e12c2ba7d6fa4d2e079c94d154c7b6d8f95bd6f: Status 404 returned error can't find the container with id 8c8c6c3358f9a3fededcca4d6e12c2ba7d6fa4d2e079c94d154c7b6d8f95bd6f
Nov 03 09:51:42.484450 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:42.484375    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:51:42.762560 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:42.762475    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c77af362-hqs7b" event={"ID":"6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7","Type":"ContainerStarted","Data":"25d57005b8aee56f4ad1d1f922e85620f1360e452bf227f02e7df6d1cc0ee443"}
Nov 03 09:51:42.762560 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:42.762538    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c77af362-hqs7b" event={"ID":"6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7","Type":"ContainerStarted","Data":"8c8c6c3358f9a3fededcca4d6e12c2ba7d6fa4d2e079c94d154c7b6d8f95bd6f"}
Nov 03 09:51:42.880992 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:42.880907    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c77af362-hqs7b" podStartSLOduration=1.8808928489999999 podStartE2EDuration="1.880892849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:51:42.878349694 +0000 UTC m=+4327.147413154" watchObservedRunningTime="2026-11-03 09:51:42.880892849 +0000 UTC m=+4327.149956282"
Nov 03 09:51:43.770151 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:43.770119    4473 generic.go:356] "Generic (PLEG): container finished" podID="6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7" containerID="25d57005b8aee56f4ad1d1f922e85620f1360e452bf227f02e7df6d1cc0ee443" exitCode=0
Nov 03 09:51:43.770540 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:43.770209    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c77af362-hqs7b" event={"ID":"6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7","Type":"ContainerDied","Data":"25d57005b8aee56f4ad1d1f922e85620f1360e452bf227f02e7df6d1cc0ee443"}
Nov 03 09:51:44.926217 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:44.926191    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c77af362-hqs7b"
Nov 03 09:51:44.988741 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:44.988711    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7-maas-api-openshift-ca\") pod \"6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7\" (UID: \"6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7\") "
Nov 03 09:51:44.988874 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:44.988857    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7-maas-api-xks-ca\") pod \"6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7\" (UID: \"6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7\") "
Nov 03 09:51:44.988924 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:44.988907    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7-kube-api-access-p9qh2\" (UniqueName: \"kubernetes.io/projected/6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7-kube-api-access-p9qh2\") pod \"6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7\" (UID: \"6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7\") "
Nov 03 09:51:44.989112 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:44.989088    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7-maas-api-openshift-ca" pod "6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7" (UID: "6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:51:44.989440 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:44.989406    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:51:44.991168 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:44.991146    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7-maas-api-xks-ca" pod "6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7" (UID: "6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:51:44.991265 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:44.991168    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7-kube-api-access-p9qh2" pod "6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7" (UID: "6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7"). InnerVolumeSpecName "kube-api-access-p9qh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:51:45.090127 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:45.090054    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:51:45.090127 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:45.090082    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9qh2\" (UniqueName: \"kubernetes.io/projected/6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7-kube-api-access-p9qh2\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:51:45.783016 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:45.782982    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c77af362-hqs7b" event={"ID":"6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7","Type":"ContainerDied","Data":"8c8c6c3358f9a3fededcca4d6e12c2ba7d6fa4d2e079c94d154c7b6d8f95bd6f"}
Nov 03 09:51:45.783194 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:45.783040    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c8c6c3358f9a3fededcca4d6e12c2ba7d6fa4d2e079c94d154c7b6d8f95bd6f"
Nov 03 09:51:45.783194 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:45.782996    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c77af362-hqs7b"
Nov 03 09:51:46.937191 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:51:46.937158    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:51:53.164857 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:53.164823    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc60feb6-2jr79"]
Nov 03 09:51:53.165369 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:53.165355    4473 cpu_manager.go:414] "containerMap: removing container" podUID="6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7" containerName="revoke-keys"
Nov 03 09:51:53.165417 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:53.165370    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7" containerName="revoke-keys"
Nov 03 09:51:53.165453 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:53.165447    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7" containerName="revoke-keys"
Nov 03 09:51:53.169182 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:53.169164    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc60feb6-2jr79"
Nov 03 09:51:53.226642 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:53.226607    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc60feb6-2jr79"]
Nov 03 09:51:53.273233 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:53.273198    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:51:53.273528 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:53.273509    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:51:53.273599 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:53.273574    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:51:53.273599 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:53.273574    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:51:53.279806 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:53.279785    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp7gb\" (UniqueName: \"kubernetes.io/projected/3adfe3ef-a39a-4536-b2f1-c2c847341b66-kube-api-access-qp7gb\") pod \"maas-api-revoke-sub-dc60feb6-2jr79\" (UID: \"3adfe3ef-a39a-4536-b2f1-c2c847341b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc60feb6-2jr79"
Nov 03 09:51:53.279891 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:53.279847    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3adfe3ef-a39a-4536-b2f1-c2c847341b66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc60feb6-2jr79\" (UID: \"3adfe3ef-a39a-4536-b2f1-c2c847341b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc60feb6-2jr79"
Nov 03 09:51:53.280043 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:53.280027    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3adfe3ef-a39a-4536-b2f1-c2c847341b66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc60feb6-2jr79\" (UID: \"3adfe3ef-a39a-4536-b2f1-c2c847341b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc60feb6-2jr79"
Nov 03 09:51:53.380844 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:53.380814    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3adfe3ef-a39a-4536-b2f1-c2c847341b66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc60feb6-2jr79\" (UID: \"3adfe3ef-a39a-4536-b2f1-c2c847341b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc60feb6-2jr79"
Nov 03 09:51:53.381026 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:53.380865    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp7gb\" (UniqueName: \"kubernetes.io/projected/3adfe3ef-a39a-4536-b2f1-c2c847341b66-kube-api-access-qp7gb\") pod \"maas-api-revoke-sub-dc60feb6-2jr79\" (UID: \"3adfe3ef-a39a-4536-b2f1-c2c847341b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc60feb6-2jr79"
Nov 03 09:51:53.381026 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:53.380920    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3adfe3ef-a39a-4536-b2f1-c2c847341b66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc60feb6-2jr79\" (UID: \"3adfe3ef-a39a-4536-b2f1-c2c847341b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc60feb6-2jr79"
Nov 03 09:51:53.381623 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:53.381594    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3adfe3ef-a39a-4536-b2f1-c2c847341b66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc60feb6-2jr79\" (UID: \"3adfe3ef-a39a-4536-b2f1-c2c847341b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc60feb6-2jr79"
Nov 03 09:51:53.383326 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:53.383307    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3adfe3ef-a39a-4536-b2f1-c2c847341b66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc60feb6-2jr79\" (UID: \"3adfe3ef-a39a-4536-b2f1-c2c847341b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc60feb6-2jr79"
Nov 03 09:51:53.439938 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:53.439879    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp7gb\" (UniqueName: \"kubernetes.io/projected/3adfe3ef-a39a-4536-b2f1-c2c847341b66-kube-api-access-qp7gb\") pod \"maas-api-revoke-sub-dc60feb6-2jr79\" (UID: \"3adfe3ef-a39a-4536-b2f1-c2c847341b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc60feb6-2jr79"
Nov 03 09:51:53.482056 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:53.482027    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc60feb6-2jr79"
Nov 03 09:51:53.861044 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:53.861010    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc60feb6-2jr79"]
Nov 03 09:51:53.870109 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:51:53.870070    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3adfe3ef_a39a_4536_b2f1_c2c847341b66.slice/crio-ac877dd3dee1c8e100ad0ee18e43ee0593f07ea85ddcb0afaa8d599efe7254dd WatchSource:0}: Error finding container ac877dd3dee1c8e100ad0ee18e43ee0593f07ea85ddcb0afaa8d599efe7254dd: Status 404 returned error can't find the container with id ac877dd3dee1c8e100ad0ee18e43ee0593f07ea85ddcb0afaa8d599efe7254dd
Nov 03 09:51:54.878107 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:54.878073    4473 generic.go:356] "Generic (PLEG): container finished" podID="3adfe3ef-a39a-4536-b2f1-c2c847341b66" containerID="e67de5123a218acada7e208b9456c5d53c053b05c96d8fa03a0f3518a70d1c23" exitCode=0
Nov 03 09:51:54.878500 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:54.878149    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc60feb6-2jr79" event={"ID":"3adfe3ef-a39a-4536-b2f1-c2c847341b66","Type":"ContainerDied","Data":"e67de5123a218acada7e208b9456c5d53c053b05c96d8fa03a0f3518a70d1c23"}
Nov 03 09:51:54.878500 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:54.878198    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc60feb6-2jr79" event={"ID":"3adfe3ef-a39a-4536-b2f1-c2c847341b66","Type":"ContainerStarted","Data":"ac877dd3dee1c8e100ad0ee18e43ee0593f07ea85ddcb0afaa8d599efe7254dd"}
Nov 03 09:51:56.024305 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:56.024281    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc60feb6-2jr79"
Nov 03 09:51:56.109647 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:56.109613    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3adfe3ef-a39a-4536-b2f1-c2c847341b66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3adfe3ef-a39a-4536-b2f1-c2c847341b66-maas-api-openshift-ca\") pod \"3adfe3ef-a39a-4536-b2f1-c2c847341b66\" (UID: \"3adfe3ef-a39a-4536-b2f1-c2c847341b66\") "
Nov 03 09:51:56.109849 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:56.109696    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3adfe3ef-a39a-4536-b2f1-c2c847341b66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3adfe3ef-a39a-4536-b2f1-c2c847341b66-maas-api-xks-ca\") pod \"3adfe3ef-a39a-4536-b2f1-c2c847341b66\" (UID: \"3adfe3ef-a39a-4536-b2f1-c2c847341b66\") "
Nov 03 09:51:56.109849 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:56.109753    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3adfe3ef-a39a-4536-b2f1-c2c847341b66-kube-api-access-qp7gb\" (UniqueName: \"kubernetes.io/projected/3adfe3ef-a39a-4536-b2f1-c2c847341b66-kube-api-access-qp7gb\") pod \"3adfe3ef-a39a-4536-b2f1-c2c847341b66\" (UID: \"3adfe3ef-a39a-4536-b2f1-c2c847341b66\") "
Nov 03 09:51:56.110057 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:56.110030    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3adfe3ef-a39a-4536-b2f1-c2c847341b66-maas-api-openshift-ca" pod "3adfe3ef-a39a-4536-b2f1-c2c847341b66" (UID: "3adfe3ef-a39a-4536-b2f1-c2c847341b66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:51:56.110345 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:56.110320    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3adfe3ef-a39a-4536-b2f1-c2c847341b66-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:51:56.113447 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:56.113415    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3adfe3ef-a39a-4536-b2f1-c2c847341b66-maas-api-xks-ca" pod "3adfe3ef-a39a-4536-b2f1-c2c847341b66" (UID: "3adfe3ef-a39a-4536-b2f1-c2c847341b66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:51:56.114780 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:56.114753    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3adfe3ef-a39a-4536-b2f1-c2c847341b66-kube-api-access-qp7gb" pod "3adfe3ef-a39a-4536-b2f1-c2c847341b66" (UID: "3adfe3ef-a39a-4536-b2f1-c2c847341b66"). InnerVolumeSpecName "kube-api-access-qp7gb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:51:56.212088 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:56.212001    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3adfe3ef-a39a-4536-b2f1-c2c847341b66-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:51:56.212088 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:56.212029    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp7gb\" (UniqueName: \"kubernetes.io/projected/3adfe3ef-a39a-4536-b2f1-c2c847341b66-kube-api-access-qp7gb\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:51:56.891346 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:56.891318    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc60feb6-2jr79"
Nov 03 09:51:56.891525 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:56.891346    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc60feb6-2jr79" event={"ID":"3adfe3ef-a39a-4536-b2f1-c2c847341b66","Type":"ContainerDied","Data":"ac877dd3dee1c8e100ad0ee18e43ee0593f07ea85ddcb0afaa8d599efe7254dd"}
Nov 03 09:51:56.891525 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:51:56.891395    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac877dd3dee1c8e100ad0ee18e43ee0593f07ea85ddcb0afaa8d599efe7254dd"
Nov 03 09:52:04.184968 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:04.165870    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18f4b584-t4dqg"]
Nov 03 09:52:04.184968 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:04.166730    4473 cpu_manager.go:414] "containerMap: removing container" podUID="3adfe3ef-a39a-4536-b2f1-c2c847341b66" containerName="revoke-keys"
Nov 03 09:52:04.184968 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:04.166751    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3adfe3ef-a39a-4536-b2f1-c2c847341b66" containerName="revoke-keys"
Nov 03 09:52:04.184968 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:04.166882    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="3adfe3ef-a39a-4536-b2f1-c2c847341b66" containerName="revoke-keys"
Nov 03 09:52:04.188104 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:04.188076    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f4b584-t4dqg"
Nov 03 09:52:04.217184 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:04.217150    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18f4b584-t4dqg"]
Nov 03 09:52:04.258672 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:04.258641    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:52:04.258865 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:04.258685    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:52:04.258865 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:04.258711    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:52:04.293470 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:04.293449    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:52:04.312069 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:04.312044    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cvv8\" (UniqueName: \"kubernetes.io/projected/8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2-kube-api-access-6cvv8\") pod \"maas-api-revoke-sub-18f4b584-t4dqg\" (UID: \"8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f4b584-t4dqg"
Nov 03 09:52:04.312231 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:04.312155    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18f4b584-t4dqg\" (UID: \"8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f4b584-t4dqg"
Nov 03 09:52:04.312231 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:04.312220    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18f4b584-t4dqg\" (UID: \"8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f4b584-t4dqg"
Nov 03 09:52:04.412993 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:04.412936    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cvv8\" (UniqueName: \"kubernetes.io/projected/8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2-kube-api-access-6cvv8\") pod \"maas-api-revoke-sub-18f4b584-t4dqg\" (UID: \"8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f4b584-t4dqg"
Nov 03 09:52:04.413182 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:04.413086    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18f4b584-t4dqg\" (UID: \"8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f4b584-t4dqg"
Nov 03 09:52:04.413251 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:04.413176    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18f4b584-t4dqg\" (UID: \"8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f4b584-t4dqg"
Nov 03 09:52:04.413882 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:04.413855    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18f4b584-t4dqg\" (UID: \"8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f4b584-t4dqg"
Nov 03 09:52:04.415659 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:04.415640    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18f4b584-t4dqg\" (UID: \"8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f4b584-t4dqg"
Nov 03 09:52:04.472167 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:04.472096    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cvv8\" (UniqueName: \"kubernetes.io/projected/8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2-kube-api-access-6cvv8\") pod \"maas-api-revoke-sub-18f4b584-t4dqg\" (UID: \"8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f4b584-t4dqg"
Nov 03 09:52:04.522011 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:04.521980    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f4b584-t4dqg"
Nov 03 09:52:04.911786 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:04.911760    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18f4b584-t4dqg"]
Nov 03 09:52:04.912108 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:52:04.912086    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d99a47e_d30d_4f87_a0a7_d3e3de0d5bd2.slice/crio-3c2ba2b1b381e5dd512aada578437f643f5cf5da362aae8d61eaff7273b76def WatchSource:0}: Error finding container 3c2ba2b1b381e5dd512aada578437f643f5cf5da362aae8d61eaff7273b76def: Status 404 returned error can't find the container with id 3c2ba2b1b381e5dd512aada578437f643f5cf5da362aae8d61eaff7273b76def
Nov 03 09:52:04.959583 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:04.959556    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f4b584-t4dqg" event={"ID":"8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2","Type":"ContainerStarted","Data":"3c2ba2b1b381e5dd512aada578437f643f5cf5da362aae8d61eaff7273b76def"}
Nov 03 09:52:05.943780 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:52:05.943750    4473 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d99a47e_d30d_4f87_a0a7_d3e3de0d5bd2.slice/crio-conmon-e9cf14080c5252e7607e780f54ae19a87aa914dcce39e6de84b62da48b86f84e.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 09:52:05.967113 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:05.967087    4473 generic.go:356] "Generic (PLEG): container finished" podID="8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2" containerID="e9cf14080c5252e7607e780f54ae19a87aa914dcce39e6de84b62da48b86f84e" exitCode=0
Nov 03 09:52:05.967276 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:05.967167    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f4b584-t4dqg" event={"ID":"8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2","Type":"ContainerDied","Data":"e9cf14080c5252e7607e780f54ae19a87aa914dcce39e6de84b62da48b86f84e"}
Nov 03 09:52:06.073841 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:06.073813    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f6de690-wc92p"]
Nov 03 09:52:06.076323 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:06.076301    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f6de690-wc92p"]
Nov 03 09:52:07.133797 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:07.133761    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f4b584-t4dqg"
Nov 03 09:52:07.242822 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:07.242738    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2-maas-api-xks-ca\") pod \"8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2\" (UID: \"8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2\") "
Nov 03 09:52:07.242822 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:07.242790    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2-kube-api-access-6cvv8\" (UniqueName: \"kubernetes.io/projected/8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2-kube-api-access-6cvv8\") pod \"8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2\" (UID: \"8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2\") "
Nov 03 09:52:07.243195 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:07.242860    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2-maas-api-openshift-ca\") pod \"8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2\" (UID: \"8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2\") "
Nov 03 09:52:07.243282 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:07.243248    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2-maas-api-openshift-ca" pod "8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2" (UID: "8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:52:07.243492 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:07.243468    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:52:07.245296 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:07.245269    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2-kube-api-access-6cvv8" pod "8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2" (UID: "8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2"). InnerVolumeSpecName "kube-api-access-6cvv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:52:07.245392 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:07.245334    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2-maas-api-xks-ca" pod "8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2" (UID: "8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:52:07.344573 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:07.344537    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:52:07.344573 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:07.344569    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cvv8\" (UniqueName: \"kubernetes.io/projected/8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2-kube-api-access-6cvv8\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:52:07.877972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:07.877904    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="448e793c-ce4a-4210-a42f-6b1ec2acbeb9" path="/var/lib/kubelet/pods/448e793c-ce4a-4210-a42f-6b1ec2acbeb9/volumes"
Nov 03 09:52:07.997244 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:07.997202    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f4b584-t4dqg" event={"ID":"8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2","Type":"ContainerDied","Data":"3c2ba2b1b381e5dd512aada578437f643f5cf5da362aae8d61eaff7273b76def"}
Nov 03 09:52:07.997244 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:07.997236    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f4b584-t4dqg"
Nov 03 09:52:07.997521 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:07.997269    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c2ba2b1b381e5dd512aada578437f643f5cf5da362aae8d61eaff7273b76def"
Nov 03 09:52:16.936726 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:52:16.936612    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:52:21.764484 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:21.764452    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:52:46.937444 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:52:46.937404    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:52:56.301125 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:52:56.301094    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:53:13.647971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:13.640218    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"]
Nov 03 09:53:13.647971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:13.640505    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt" podUID="5a6ab0a4-33a9-43f4-975b-2e3a6261e300" containerName="istio-proxy" containerID="cri-o://2127eea8f47096b95545a7a0219a195ae3bd8f1fab1eefe055e519fead578408" gracePeriod=30
Nov 03 09:53:13.696271 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:13.696227    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"]
Nov 03 09:53:13.696843 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:13.696807    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh" podUID="7e4c0879-52a1-4038-aedd-3810b70a76d6" containerName="istio-proxy" containerID="cri-o://c05c2fa8c1d5bd139ecf0972bd2afd405dbb16727f6978bd7d92260ff27bf6ab" gracePeriod=30
Nov 03 09:53:15.795830 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:15.795788    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-wchmh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 09:53:15.796238 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:15.795860    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh" podUID="7e4c0879-52a1-4038-aedd-3810b70a76d6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 09:53:15.884255 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:15.884223    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 09:53:15.884427 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:15.884279    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt" podUID="5a6ab0a4-33a9-43f4-975b-2e3a6261e300" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 09:53:16.937463 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:53:16.937426    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:53:18.981329 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:18.981269    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"
Nov 03 09:53:19.004020 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.004000    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"
Nov 03 09:53:19.080407 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.080379    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e4c0879-52a1-4038-aedd-3810b70a76d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e4c0879-52a1-4038-aedd-3810b70a76d6-istiod-ca-cert\") pod \"7e4c0879-52a1-4038-aedd-3810b70a76d6\" (UID: \"7e4c0879-52a1-4038-aedd-3810b70a76d6\") "
Nov 03 09:53:19.080407 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.080413    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-istiod-ca-cert\") pod \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\" (UID: \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\") "
Nov 03 09:53:19.080672 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.080434    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e4c0879-52a1-4038-aedd-3810b70a76d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0879-52a1-4038-aedd-3810b70a76d6-workload-certs\") pod \"7e4c0879-52a1-4038-aedd-3810b70a76d6\" (UID: \"7e4c0879-52a1-4038-aedd-3810b70a76d6\") "
Nov 03 09:53:19.080672 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.080509    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e4c0879-52a1-4038-aedd-3810b70a76d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0879-52a1-4038-aedd-3810b70a76d6-istio-envoy\") pod \"7e4c0879-52a1-4038-aedd-3810b70a76d6\" (UID: \"7e4c0879-52a1-4038-aedd-3810b70a76d6\") "
Nov 03 09:53:19.080672 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.080532    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e4c0879-52a1-4038-aedd-3810b70a76d6-istio-token\" (UniqueName: \"kubernetes.io/projected/7e4c0879-52a1-4038-aedd-3810b70a76d6-istio-token\") pod \"7e4c0879-52a1-4038-aedd-3810b70a76d6\" (UID: \"7e4c0879-52a1-4038-aedd-3810b70a76d6\") "
Nov 03 09:53:19.080672 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.080561    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-kube-api-access-hcsrl\" (UniqueName: \"kubernetes.io/projected/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-kube-api-access-hcsrl\") pod \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\" (UID: \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\") "
Nov 03 09:53:19.080672 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.080612    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e4c0879-52a1-4038-aedd-3810b70a76d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0879-52a1-4038-aedd-3810b70a76d6-istio-data\") pod \"7e4c0879-52a1-4038-aedd-3810b70a76d6\" (UID: \"7e4c0879-52a1-4038-aedd-3810b70a76d6\") "
Nov 03 09:53:19.080672 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.080646    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e4c0879-52a1-4038-aedd-3810b70a76d6-kube-api-access-cm8xh\" (UniqueName: \"kubernetes.io/projected/7e4c0879-52a1-4038-aedd-3810b70a76d6-kube-api-access-cm8xh\") pod \"7e4c0879-52a1-4038-aedd-3810b70a76d6\" (UID: \"7e4c0879-52a1-4038-aedd-3810b70a76d6\") "
Nov 03 09:53:19.080672 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.080675    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-istio-podinfo\") pod \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\" (UID: \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\") "
Nov 03 09:53:19.081103 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.080723    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e4c0879-52a1-4038-aedd-3810b70a76d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0879-52a1-4038-aedd-3810b70a76d6-credential-socket\") pod \"7e4c0879-52a1-4038-aedd-3810b70a76d6\" (UID: \"7e4c0879-52a1-4038-aedd-3810b70a76d6\") "
Nov 03 09:53:19.081103 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.080752    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-istio-envoy\") pod \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\" (UID: \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\") "
Nov 03 09:53:19.081103 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.080785    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4c0879-52a1-4038-aedd-3810b70a76d6-workload-certs" pod "7e4c0879-52a1-4038-aedd-3810b70a76d6" (UID: "7e4c0879-52a1-4038-aedd-3810b70a76d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:53:19.081103 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.080810    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-istio-token\" (UniqueName: \"kubernetes.io/projected/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-istio-token\") pod \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\" (UID: \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\") "
Nov 03 09:53:19.081103 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.080832    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e4c0879-52a1-4038-aedd-3810b70a76d6-istiod-ca-cert" pod "7e4c0879-52a1-4038-aedd-3810b70a76d6" (UID: "7e4c0879-52a1-4038-aedd-3810b70a76d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:53:19.081103 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.080920    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4c0879-52a1-4038-aedd-3810b70a76d6-istio-data" pod "7e4c0879-52a1-4038-aedd-3810b70a76d6" (UID: "7e4c0879-52a1-4038-aedd-3810b70a76d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:53:19.081103 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.080854    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-workload-socket\") pod \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\" (UID: \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\") "
Nov 03 09:53:19.081103 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.081002    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-istiod-ca-cert" pod "5a6ab0a4-33a9-43f4-975b-2e3a6261e300" (UID: "5a6ab0a4-33a9-43f4-975b-2e3a6261e300"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:53:19.081103 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.081076    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-credential-socket\") pod \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\" (UID: \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\") "
Nov 03 09:53:19.081774 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.081101    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-workload-socket" pod "5a6ab0a4-33a9-43f4-975b-2e3a6261e300" (UID: "5a6ab0a4-33a9-43f4-975b-2e3a6261e300"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:53:19.081774 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.081107    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-istio-data\") pod \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\" (UID: \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\") "
Nov 03 09:53:19.081774 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.081164    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e4c0879-52a1-4038-aedd-3810b70a76d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0879-52a1-4038-aedd-3810b70a76d6-workload-socket\") pod \"7e4c0879-52a1-4038-aedd-3810b70a76d6\" (UID: \"7e4c0879-52a1-4038-aedd-3810b70a76d6\") "
Nov 03 09:53:19.081774 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.081197    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e4c0879-52a1-4038-aedd-3810b70a76d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e4c0879-52a1-4038-aedd-3810b70a76d6-istio-podinfo\") pod \"7e4c0879-52a1-4038-aedd-3810b70a76d6\" (UID: \"7e4c0879-52a1-4038-aedd-3810b70a76d6\") "
Nov 03 09:53:19.081774 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.081289    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-istio-data" pod "5a6ab0a4-33a9-43f4-975b-2e3a6261e300" (UID: "5a6ab0a4-33a9-43f4-975b-2e3a6261e300"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:53:19.081774 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.081304    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-workload-certs\") pod \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\" (UID: \"5a6ab0a4-33a9-43f4-975b-2e3a6261e300\") "
Nov 03 09:53:19.081774 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.081633    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-credential-socket" pod "5a6ab0a4-33a9-43f4-975b-2e3a6261e300" (UID: "5a6ab0a4-33a9-43f4-975b-2e3a6261e300"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:53:19.082546 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.081797    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4c0879-52a1-4038-aedd-3810b70a76d6-credential-socket" pod "7e4c0879-52a1-4038-aedd-3810b70a76d6" (UID: "7e4c0879-52a1-4038-aedd-3810b70a76d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:53:19.082546 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.081969    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4c0879-52a1-4038-aedd-3810b70a76d6-workload-socket" pod "7e4c0879-52a1-4038-aedd-3810b70a76d6" (UID: "7e4c0879-52a1-4038-aedd-3810b70a76d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:53:19.082546 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.082052    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-workload-certs" pod "5a6ab0a4-33a9-43f4-975b-2e3a6261e300" (UID: "5a6ab0a4-33a9-43f4-975b-2e3a6261e300"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:53:19.082546 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.082128    4473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0879-52a1-4038-aedd-3810b70a76d6-istio-data\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:53:19.082546 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.082149    4473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0879-52a1-4038-aedd-3810b70a76d6-credential-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:53:19.082546 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.082165    4473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-workload-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:53:19.082546 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.082178    4473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-credential-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:53:19.082546 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.082192    4473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-istio-data\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:53:19.082546 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.082206    4473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0879-52a1-4038-aedd-3810b70a76d6-workload-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:53:19.082546 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.082221    4473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e4c0879-52a1-4038-aedd-3810b70a76d6-istiod-ca-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:53:19.082546 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.082235    4473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-istiod-ca-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:53:19.082546 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.082249    4473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0879-52a1-4038-aedd-3810b70a76d6-workload-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:53:19.083710 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.083679    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-istio-envoy" pod "5a6ab0a4-33a9-43f4-975b-2e3a6261e300" (UID: "5a6ab0a4-33a9-43f4-975b-2e3a6261e300"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:53:19.083710 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.083696    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-istio-podinfo" pod "5a6ab0a4-33a9-43f4-975b-2e3a6261e300" (UID: "5a6ab0a4-33a9-43f4-975b-2e3a6261e300"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 09:53:19.083837 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.083754    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-kube-api-access-hcsrl" pod "5a6ab0a4-33a9-43f4-975b-2e3a6261e300" (UID: "5a6ab0a4-33a9-43f4-975b-2e3a6261e300"). InnerVolumeSpecName "kube-api-access-hcsrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:53:19.083837 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.083805    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e4c0879-52a1-4038-aedd-3810b70a76d6-kube-api-access-cm8xh" pod "7e4c0879-52a1-4038-aedd-3810b70a76d6" (UID: "7e4c0879-52a1-4038-aedd-3810b70a76d6"). InnerVolumeSpecName "kube-api-access-cm8xh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:53:19.084395 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.084377    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e4c0879-52a1-4038-aedd-3810b70a76d6-istio-podinfo" pod "7e4c0879-52a1-4038-aedd-3810b70a76d6" (UID: "7e4c0879-52a1-4038-aedd-3810b70a76d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 09:53:19.084803 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.084781    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-istio-token" pod "5a6ab0a4-33a9-43f4-975b-2e3a6261e300" (UID: "5a6ab0a4-33a9-43f4-975b-2e3a6261e300"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:53:19.085279 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.085262    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e4c0879-52a1-4038-aedd-3810b70a76d6-istio-token" pod "7e4c0879-52a1-4038-aedd-3810b70a76d6" (UID: "7e4c0879-52a1-4038-aedd-3810b70a76d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:53:19.085279 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.085268    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e4c0879-52a1-4038-aedd-3810b70a76d6-istio-envoy" pod "7e4c0879-52a1-4038-aedd-3810b70a76d6" (UID: "7e4c0879-52a1-4038-aedd-3810b70a76d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:53:19.183536 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.183495    4473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e4c0879-52a1-4038-aedd-3810b70a76d6-istio-podinfo\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:53:19.183536 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.183529    4473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-workload-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:53:19.183536 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.183542    4473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e4c0879-52a1-4038-aedd-3810b70a76d6-istio-envoy\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:53:19.183781 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.183554    4473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e4c0879-52a1-4038-aedd-3810b70a76d6-istio-token\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:53:19.183781 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.183567    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcsrl\" (UniqueName: \"kubernetes.io/projected/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-kube-api-access-hcsrl\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:53:19.183781 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.183580    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm8xh\" (UniqueName: \"kubernetes.io/projected/7e4c0879-52a1-4038-aedd-3810b70a76d6-kube-api-access-cm8xh\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:53:19.183781 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.183591    4473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-istio-podinfo\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:53:19.183781 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.183603    4473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-istio-envoy\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:53:19.183781 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.183614    4473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a6ab0a4-33a9-43f4-975b-2e3a6261e300-istio-token\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:53:19.463697 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.463657    4473 generic.go:356] "Generic (PLEG): container finished" podID="5a6ab0a4-33a9-43f4-975b-2e3a6261e300" containerID="2127eea8f47096b95545a7a0219a195ae3bd8f1fab1eefe055e519fead578408" exitCode=0
Nov 03 09:53:19.464356 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.463694    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt" event={"ID":"5a6ab0a4-33a9-43f4-975b-2e3a6261e300","Type":"ContainerDied","Data":"2127eea8f47096b95545a7a0219a195ae3bd8f1fab1eefe055e519fead578408"}
Nov 03 09:53:19.464356 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.463750    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"
Nov 03 09:53:19.464356 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.463779    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt" event={"ID":"5a6ab0a4-33a9-43f4-975b-2e3a6261e300","Type":"ContainerDied","Data":"f9ef2add721f1182d8b32f7407ec37583b16408bc6b151a67458749f408514f8"}
Nov 03 09:53:19.464356 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.463803    4473 scope.go:122] "RemoveContainer" containerID="2127eea8f47096b95545a7a0219a195ae3bd8f1fab1eefe055e519fead578408"
Nov 03 09:53:19.465495 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.465467    4473 generic.go:356] "Generic (PLEG): container finished" podID="7e4c0879-52a1-4038-aedd-3810b70a76d6" containerID="c05c2fa8c1d5bd139ecf0972bd2afd405dbb16727f6978bd7d92260ff27bf6ab" exitCode=0
Nov 03 09:53:19.465625 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.465541    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh" event={"ID":"7e4c0879-52a1-4038-aedd-3810b70a76d6","Type":"ContainerDied","Data":"c05c2fa8c1d5bd139ecf0972bd2afd405dbb16727f6978bd7d92260ff27bf6ab"}
Nov 03 09:53:19.465625 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.465584    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh" event={"ID":"7e4c0879-52a1-4038-aedd-3810b70a76d6","Type":"ContainerDied","Data":"7f8fdf7b6459407f8f3be4dd2c44f28f702d03311b65ff458272e392a0eef045"}
Nov 03 09:53:19.465757 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.465666    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"
Nov 03 09:53:19.476259 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.476241    4473 scope.go:122] "RemoveContainer" containerID="2127eea8f47096b95545a7a0219a195ae3bd8f1fab1eefe055e519fead578408"
Nov 03 09:53:19.476576 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:53:19.476555    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2127eea8f47096b95545a7a0219a195ae3bd8f1fab1eefe055e519fead578408\": container with ID starting with 2127eea8f47096b95545a7a0219a195ae3bd8f1fab1eefe055e519fead578408 not found: ID does not exist" containerID="2127eea8f47096b95545a7a0219a195ae3bd8f1fab1eefe055e519fead578408"
Nov 03 09:53:19.476639 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.476588    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2127eea8f47096b95545a7a0219a195ae3bd8f1fab1eefe055e519fead578408"} err="failed to get container status \"2127eea8f47096b95545a7a0219a195ae3bd8f1fab1eefe055e519fead578408\": rpc error: code = NotFound desc = could not find container \"2127eea8f47096b95545a7a0219a195ae3bd8f1fab1eefe055e519fead578408\": container with ID starting with 2127eea8f47096b95545a7a0219a195ae3bd8f1fab1eefe055e519fead578408 not found: ID does not exist"
Nov 03 09:53:19.476639 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.476612    4473 scope.go:122] "RemoveContainer" containerID="c05c2fa8c1d5bd139ecf0972bd2afd405dbb16727f6978bd7d92260ff27bf6ab"
Nov 03 09:53:19.486518 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.486500    4473 scope.go:122] "RemoveContainer" containerID="c05c2fa8c1d5bd139ecf0972bd2afd405dbb16727f6978bd7d92260ff27bf6ab"
Nov 03 09:53:19.486749 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:53:19.486731    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c05c2fa8c1d5bd139ecf0972bd2afd405dbb16727f6978bd7d92260ff27bf6ab\": container with ID starting with c05c2fa8c1d5bd139ecf0972bd2afd405dbb16727f6978bd7d92260ff27bf6ab not found: ID does not exist" containerID="c05c2fa8c1d5bd139ecf0972bd2afd405dbb16727f6978bd7d92260ff27bf6ab"
Nov 03 09:53:19.486803 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.486753    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c05c2fa8c1d5bd139ecf0972bd2afd405dbb16727f6978bd7d92260ff27bf6ab"} err="failed to get container status \"c05c2fa8c1d5bd139ecf0972bd2afd405dbb16727f6978bd7d92260ff27bf6ab\": rpc error: code = NotFound desc = could not find container \"c05c2fa8c1d5bd139ecf0972bd2afd405dbb16727f6978bd7d92260ff27bf6ab\": container with ID starting with c05c2fa8c1d5bd139ecf0972bd2afd405dbb16727f6978bd7d92260ff27bf6ab not found: ID does not exist"
Nov 03 09:53:19.633508 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.633472    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"]
Nov 03 09:53:19.638643 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.638609    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jfwbt"]
Nov 03 09:53:19.750633 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.750604    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"]
Nov 03 09:53:19.756802 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.756771    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-wchmh"]
Nov 03 09:53:19.872520 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.872488    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a6ab0a4-33a9-43f4-975b-2e3a6261e300" path="/var/lib/kubelet/pods/5a6ab0a4-33a9-43f4-975b-2e3a6261e300/volumes"
Nov 03 09:53:19.872990 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:19.872975    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e4c0879-52a1-4038-aedd-3810b70a76d6" path="/var/lib/kubelet/pods/7e4c0879-52a1-4038-aedd-3810b70a76d6/volumes"
Nov 03 09:53:28.884700 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:28.884657    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:53:33.250516 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:33.250483    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af909b33-5cnmx"]
Nov 03 09:53:33.251007 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:33.250976    4473 cpu_manager.go:414] "containerMap: removing container" podUID="7e4c0879-52a1-4038-aedd-3810b70a76d6" containerName="istio-proxy"
Nov 03 09:53:33.251007 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:33.250990    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e4c0879-52a1-4038-aedd-3810b70a76d6" containerName="istio-proxy"
Nov 03 09:53:33.251103 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:33.251012    4473 cpu_manager.go:414] "containerMap: removing container" podUID="5a6ab0a4-33a9-43f4-975b-2e3a6261e300" containerName="istio-proxy"
Nov 03 09:53:33.251103 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:33.251019    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a6ab0a4-33a9-43f4-975b-2e3a6261e300" containerName="istio-proxy"
Nov 03 09:53:33.251103 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:33.251043    4473 cpu_manager.go:414] "containerMap: removing container" podUID="8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2" containerName="revoke-keys"
Nov 03 09:53:33.251103 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:33.251048    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2" containerName="revoke-keys"
Nov 03 09:53:33.251236 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:33.251123    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2" containerName="revoke-keys"
Nov 03 09:53:33.251236 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:33.251134    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e4c0879-52a1-4038-aedd-3810b70a76d6" containerName="istio-proxy"
Nov 03 09:53:33.251236 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:33.251142    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a6ab0a4-33a9-43f4-975b-2e3a6261e300" containerName="istio-proxy"
Nov 03 09:53:33.256098 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:33.256077    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af909b33-5cnmx"
Nov 03 09:53:33.312271 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:33.312162    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af909b33-5cnmx"]
Nov 03 09:53:33.359033 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:33.358999    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:53:33.359033 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:33.359003    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:53:33.359033 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:33.359012    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:53:33.360095 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:33.360075    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:53:33.439155 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:33.439125    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnc82\" (UniqueName: \"kubernetes.io/projected/1464a718-105e-4039-b0be-b7955701e38d-kube-api-access-nnc82\") pod \"maas-api-revoke-sub-af909b33-5cnmx\" (UID: \"1464a718-105e-4039-b0be-b7955701e38d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af909b33-5cnmx"
Nov 03 09:53:33.439299 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:33.439245    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1464a718-105e-4039-b0be-b7955701e38d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af909b33-5cnmx\" (UID: \"1464a718-105e-4039-b0be-b7955701e38d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af909b33-5cnmx"
Nov 03 09:53:33.439299 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:33.439278    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1464a718-105e-4039-b0be-b7955701e38d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af909b33-5cnmx\" (UID: \"1464a718-105e-4039-b0be-b7955701e38d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af909b33-5cnmx"
Nov 03 09:53:33.540555 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:33.540472    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnc82\" (UniqueName: \"kubernetes.io/projected/1464a718-105e-4039-b0be-b7955701e38d-kube-api-access-nnc82\") pod \"maas-api-revoke-sub-af909b33-5cnmx\" (UID: \"1464a718-105e-4039-b0be-b7955701e38d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af909b33-5cnmx"
Nov 03 09:53:33.540722 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:33.540598    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1464a718-105e-4039-b0be-b7955701e38d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af909b33-5cnmx\" (UID: \"1464a718-105e-4039-b0be-b7955701e38d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af909b33-5cnmx"
Nov 03 09:53:33.540722 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:33.540629    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1464a718-105e-4039-b0be-b7955701e38d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af909b33-5cnmx\" (UID: \"1464a718-105e-4039-b0be-b7955701e38d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af909b33-5cnmx"
Nov 03 09:53:33.541278 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:33.541258    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1464a718-105e-4039-b0be-b7955701e38d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af909b33-5cnmx\" (UID: \"1464a718-105e-4039-b0be-b7955701e38d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af909b33-5cnmx"
Nov 03 09:53:33.543177 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:33.543158    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1464a718-105e-4039-b0be-b7955701e38d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af909b33-5cnmx\" (UID: \"1464a718-105e-4039-b0be-b7955701e38d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af909b33-5cnmx"
Nov 03 09:53:33.602903 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:33.602873    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnc82\" (UniqueName: \"kubernetes.io/projected/1464a718-105e-4039-b0be-b7955701e38d-kube-api-access-nnc82\") pod \"maas-api-revoke-sub-af909b33-5cnmx\" (UID: \"1464a718-105e-4039-b0be-b7955701e38d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af909b33-5cnmx"
Nov 03 09:53:33.869249 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:33.869225    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af909b33-5cnmx"
Nov 03 09:53:34.248915 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:34.248890    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af909b33-5cnmx"]
Nov 03 09:53:34.250888 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:53:34.250857    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1464a718_105e_4039_b0be_b7955701e38d.slice/crio-39047b14cded5012666c4e190724f5ac26151598d1cefcd87724473a2208cb99 WatchSource:0}: Error finding container 39047b14cded5012666c4e190724f5ac26151598d1cefcd87724473a2208cb99: Status 404 returned error can't find the container with id 39047b14cded5012666c4e190724f5ac26151598d1cefcd87724473a2208cb99
Nov 03 09:53:34.560146 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:34.560111    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af909b33-5cnmx" event={"ID":"1464a718-105e-4039-b0be-b7955701e38d","Type":"ContainerStarted","Data":"57494d409a83b6be5506740f548a47da39160bc7800b5cd7e52f493777b33a95"}
Nov 03 09:53:34.560275 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:34.560171    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af909b33-5cnmx" event={"ID":"1464a718-105e-4039-b0be-b7955701e38d","Type":"ContainerStarted","Data":"39047b14cded5012666c4e190724f5ac26151598d1cefcd87724473a2208cb99"}
Nov 03 09:53:34.690066 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:34.690005    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af909b33-5cnmx" podStartSLOduration=1.689986325 podStartE2EDuration="1.689986325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:53:34.688240613 +0000 UTC m=+4438.957304073" watchObservedRunningTime="2026-11-03 09:53:34.689986325 +0000 UTC m=+4438.959049759"
Nov 03 09:53:35.567541 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:35.567505    4473 generic.go:356] "Generic (PLEG): container finished" podID="1464a718-105e-4039-b0be-b7955701e38d" containerID="57494d409a83b6be5506740f548a47da39160bc7800b5cd7e52f493777b33a95" exitCode=0
Nov 03 09:53:35.567903 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:35.567596    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af909b33-5cnmx" event={"ID":"1464a718-105e-4039-b0be-b7955701e38d","Type":"ContainerDied","Data":"57494d409a83b6be5506740f548a47da39160bc7800b5cd7e52f493777b33a95"}
Nov 03 09:53:36.716631 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:36.716609    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af909b33-5cnmx"
Nov 03 09:53:36.880341 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:36.880310    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1464a718-105e-4039-b0be-b7955701e38d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1464a718-105e-4039-b0be-b7955701e38d-maas-api-openshift-ca\") pod \"1464a718-105e-4039-b0be-b7955701e38d\" (UID: \"1464a718-105e-4039-b0be-b7955701e38d\") "
Nov 03 09:53:36.880532 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:36.880349    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1464a718-105e-4039-b0be-b7955701e38d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1464a718-105e-4039-b0be-b7955701e38d-maas-api-xks-ca\") pod \"1464a718-105e-4039-b0be-b7955701e38d\" (UID: \"1464a718-105e-4039-b0be-b7955701e38d\") "
Nov 03 09:53:36.880532 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:36.880437    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1464a718-105e-4039-b0be-b7955701e38d-kube-api-access-nnc82\" (UniqueName: \"kubernetes.io/projected/1464a718-105e-4039-b0be-b7955701e38d-kube-api-access-nnc82\") pod \"1464a718-105e-4039-b0be-b7955701e38d\" (UID: \"1464a718-105e-4039-b0be-b7955701e38d\") "
Nov 03 09:53:36.880658 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:36.880636    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1464a718-105e-4039-b0be-b7955701e38d-maas-api-openshift-ca" pod "1464a718-105e-4039-b0be-b7955701e38d" (UID: "1464a718-105e-4039-b0be-b7955701e38d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:53:36.880846 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:36.880827    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1464a718-105e-4039-b0be-b7955701e38d-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:53:36.882706 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:36.882678    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1464a718-105e-4039-b0be-b7955701e38d-kube-api-access-nnc82" pod "1464a718-105e-4039-b0be-b7955701e38d" (UID: "1464a718-105e-4039-b0be-b7955701e38d"). InnerVolumeSpecName "kube-api-access-nnc82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:53:36.882815 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:36.882736    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1464a718-105e-4039-b0be-b7955701e38d-maas-api-xks-ca" pod "1464a718-105e-4039-b0be-b7955701e38d" (UID: "1464a718-105e-4039-b0be-b7955701e38d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:53:36.982309 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:36.982280    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1464a718-105e-4039-b0be-b7955701e38d-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:53:36.982309 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:36.982308    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnc82\" (UniqueName: \"kubernetes.io/projected/1464a718-105e-4039-b0be-b7955701e38d-kube-api-access-nnc82\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:53:37.580369 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:37.580340    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af909b33-5cnmx"
Nov 03 09:53:37.580557 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:37.580379    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af909b33-5cnmx" event={"ID":"1464a718-105e-4039-b0be-b7955701e38d","Type":"ContainerDied","Data":"39047b14cded5012666c4e190724f5ac26151598d1cefcd87724473a2208cb99"}
Nov 03 09:53:37.580557 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:37.580456    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39047b14cded5012666c4e190724f5ac26151598d1cefcd87724473a2208cb99"
Nov 03 09:53:38.951263 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:38.951236    4473 scope.go:122] "RemoveContainer" containerID="fb0d377bc6346095de61daf082dcb7aaab33118f7ab208deada2574f2cd39aa9"
Nov 03 09:53:43.954199 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:43.954170    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d8d638a-zkz5t"]
Nov 03 09:53:43.954663 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:43.954650    4473 cpu_manager.go:414] "containerMap: removing container" podUID="1464a718-105e-4039-b0be-b7955701e38d" containerName="revoke-keys"
Nov 03 09:53:43.954704 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:43.954665    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1464a718-105e-4039-b0be-b7955701e38d" containerName="revoke-keys"
Nov 03 09:53:43.954761 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:43.954752    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="1464a718-105e-4039-b0be-b7955701e38d" containerName="revoke-keys"
Nov 03 09:53:43.959405 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:43.959386    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8d638a-zkz5t"
Nov 03 09:53:44.015699 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:44.015672    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d8d638a-zkz5t"]
Nov 03 09:53:44.052858 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:44.052831    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7528\" (UniqueName: \"kubernetes.io/projected/5e1c80a4-7dd5-4341-b7a6-b534d026b54c-kube-api-access-g7528\") pod \"maas-api-revoke-sub-4d8d638a-zkz5t\" (UID: \"5e1c80a4-7dd5-4341-b7a6-b534d026b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8d638a-zkz5t"
Nov 03 09:53:44.053026 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:44.052870    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e1c80a4-7dd5-4341-b7a6-b534d026b54c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d8d638a-zkz5t\" (UID: \"5e1c80a4-7dd5-4341-b7a6-b534d026b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8d638a-zkz5t"
Nov 03 09:53:44.053070 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:44.053036    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e1c80a4-7dd5-4341-b7a6-b534d026b54c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d8d638a-zkz5t\" (UID: \"5e1c80a4-7dd5-4341-b7a6-b534d026b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8d638a-zkz5t"
Nov 03 09:53:44.062495 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:44.062474    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:53:44.062569 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:44.062496    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:53:44.062569 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:44.062479    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:53:44.063381 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:44.063363    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:53:44.153809 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:44.153786    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e1c80a4-7dd5-4341-b7a6-b534d026b54c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d8d638a-zkz5t\" (UID: \"5e1c80a4-7dd5-4341-b7a6-b534d026b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8d638a-zkz5t"
Nov 03 09:53:44.153975 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:44.153851    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7528\" (UniqueName: \"kubernetes.io/projected/5e1c80a4-7dd5-4341-b7a6-b534d026b54c-kube-api-access-g7528\") pod \"maas-api-revoke-sub-4d8d638a-zkz5t\" (UID: \"5e1c80a4-7dd5-4341-b7a6-b534d026b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8d638a-zkz5t"
Nov 03 09:53:44.153975 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:44.153880    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e1c80a4-7dd5-4341-b7a6-b534d026b54c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d8d638a-zkz5t\" (UID: \"5e1c80a4-7dd5-4341-b7a6-b534d026b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8d638a-zkz5t"
Nov 03 09:53:44.154588 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:44.154564    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e1c80a4-7dd5-4341-b7a6-b534d026b54c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d8d638a-zkz5t\" (UID: \"5e1c80a4-7dd5-4341-b7a6-b534d026b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8d638a-zkz5t"
Nov 03 09:53:44.156296 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:44.156277    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e1c80a4-7dd5-4341-b7a6-b534d026b54c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d8d638a-zkz5t\" (UID: \"5e1c80a4-7dd5-4341-b7a6-b534d026b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8d638a-zkz5t"
Nov 03 09:53:44.213267 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:44.213203    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7528\" (UniqueName: \"kubernetes.io/projected/5e1c80a4-7dd5-4341-b7a6-b534d026b54c-kube-api-access-g7528\") pod \"maas-api-revoke-sub-4d8d638a-zkz5t\" (UID: \"5e1c80a4-7dd5-4341-b7a6-b534d026b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8d638a-zkz5t"
Nov 03 09:53:44.259938 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:44.259908    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f8439f5b-64dd678df-22vnq"]
Nov 03 09:53:44.272983 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:44.272938    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8d638a-zkz5t"
Nov 03 09:53:44.373236 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:44.373203    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f8439f5b-58b76f4b8c-x2dl8"]
Nov 03 09:53:44.649780 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:44.649753    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d8d638a-zkz5t"]
Nov 03 09:53:44.650398 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:53:44.650370    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e1c80a4_7dd5_4341_b7a6_b534d026b54c.slice/crio-f0b50ad7ed488565385b189540c55bc7429d37163a2fabfd498199fb3a9fbb19 WatchSource:0}: Error finding container f0b50ad7ed488565385b189540c55bc7429d37163a2fabfd498199fb3a9fbb19: Status 404 returned error can't find the container with id f0b50ad7ed488565385b189540c55bc7429d37163a2fabfd498199fb3a9fbb19
Nov 03 09:53:45.464669 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:45.464636    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df73bc75-zl9lt"]
Nov 03 09:53:45.470478 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:45.470459    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df73bc75-zl9lt"
Nov 03 09:53:45.532394 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:45.532366    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df73bc75-zl9lt"]
Nov 03 09:53:45.569917 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:45.569893    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm8bm\" (UniqueName: \"kubernetes.io/projected/3e87c013-ff41-4774-8e96-838df05ecc08-kube-api-access-gm8bm\") pod \"maas-api-revoke-sub-df73bc75-zl9lt\" (UID: \"3e87c013-ff41-4774-8e96-838df05ecc08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df73bc75-zl9lt"
Nov 03 09:53:45.570117 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:45.570036    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e87c013-ff41-4774-8e96-838df05ecc08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df73bc75-zl9lt\" (UID: \"3e87c013-ff41-4774-8e96-838df05ecc08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df73bc75-zl9lt"
Nov 03 09:53:45.570117 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:45.570065    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e87c013-ff41-4774-8e96-838df05ecc08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df73bc75-zl9lt\" (UID: \"3e87c013-ff41-4774-8e96-838df05ecc08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df73bc75-zl9lt"
Nov 03 09:53:45.629519 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:45.629489    4473 generic.go:356] "Generic (PLEG): container finished" podID="5e1c80a4-7dd5-4341-b7a6-b534d026b54c" containerID="8af805c6572d7a5528eec711875db25d1a976847c5babc96fa8ad02ced3fe548" exitCode=0
Nov 03 09:53:45.629640 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:45.629583    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8d638a-zkz5t" event={"ID":"5e1c80a4-7dd5-4341-b7a6-b534d026b54c","Type":"ContainerDied","Data":"8af805c6572d7a5528eec711875db25d1a976847c5babc96fa8ad02ced3fe548"}
Nov 03 09:53:45.629687 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:45.629644    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8d638a-zkz5t" event={"ID":"5e1c80a4-7dd5-4341-b7a6-b534d026b54c","Type":"ContainerStarted","Data":"f0b50ad7ed488565385b189540c55bc7429d37163a2fabfd498199fb3a9fbb19"}
Nov 03 09:53:45.671436 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:45.671411    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm8bm\" (UniqueName: \"kubernetes.io/projected/3e87c013-ff41-4774-8e96-838df05ecc08-kube-api-access-gm8bm\") pod \"maas-api-revoke-sub-df73bc75-zl9lt\" (UID: \"3e87c013-ff41-4774-8e96-838df05ecc08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df73bc75-zl9lt"
Nov 03 09:53:45.671586 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:45.671491    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e87c013-ff41-4774-8e96-838df05ecc08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df73bc75-zl9lt\" (UID: \"3e87c013-ff41-4774-8e96-838df05ecc08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df73bc75-zl9lt"
Nov 03 09:53:45.671586 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:45.671520    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e87c013-ff41-4774-8e96-838df05ecc08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df73bc75-zl9lt\" (UID: \"3e87c013-ff41-4774-8e96-838df05ecc08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df73bc75-zl9lt"
Nov 03 09:53:45.672376 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:45.672352    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e87c013-ff41-4774-8e96-838df05ecc08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df73bc75-zl9lt\" (UID: \"3e87c013-ff41-4774-8e96-838df05ecc08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df73bc75-zl9lt"
Nov 03 09:53:45.674069 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:45.674052    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e87c013-ff41-4774-8e96-838df05ecc08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df73bc75-zl9lt\" (UID: \"3e87c013-ff41-4774-8e96-838df05ecc08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df73bc75-zl9lt"
Nov 03 09:53:45.731479 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:45.731400    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm8bm\" (UniqueName: \"kubernetes.io/projected/3e87c013-ff41-4774-8e96-838df05ecc08-kube-api-access-gm8bm\") pod \"maas-api-revoke-sub-df73bc75-zl9lt\" (UID: \"3e87c013-ff41-4774-8e96-838df05ecc08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df73bc75-zl9lt"
Nov 03 09:53:45.783595 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:45.783565    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df73bc75-zl9lt"
Nov 03 09:53:46.172656 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:46.166733    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df73bc75-zl9lt"]
Nov 03 09:53:46.637247 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:46.637207    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df73bc75-zl9lt" event={"ID":"3e87c013-ff41-4774-8e96-838df05ecc08","Type":"ContainerStarted","Data":"92f683f583b52c9933e2a325bbfd42b183218fe711e8a5f758e91ae962c110e7"}
Nov 03 09:53:46.637628 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:46.637292    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df73bc75-zl9lt" event={"ID":"3e87c013-ff41-4774-8e96-838df05ecc08","Type":"ContainerStarted","Data":"62f0ac22e85cb9498f452952380722f2314e2630cac9dcd1ad148f6146efe232"}
Nov 03 09:53:46.754002 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:46.753909    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df73bc75-zl9lt" podStartSLOduration=1.753896009 podStartE2EDuration="1.753896009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:53:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:53:46.75228686 +0000 UTC m=+4451.021350312" watchObservedRunningTime="2026-11-03 09:53:46.753896009 +0000 UTC m=+4451.022959444"
Nov 03 09:53:46.772825 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:46.772800    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8d638a-zkz5t"
Nov 03 09:53:46.884785 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:46.884756    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e1c80a4-7dd5-4341-b7a6-b534d026b54c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e1c80a4-7dd5-4341-b7a6-b534d026b54c-maas-api-xks-ca\") pod \"5e1c80a4-7dd5-4341-b7a6-b534d026b54c\" (UID: \"5e1c80a4-7dd5-4341-b7a6-b534d026b54c\") "
Nov 03 09:53:46.884996 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:46.884863    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e1c80a4-7dd5-4341-b7a6-b534d026b54c-kube-api-access-g7528\" (UniqueName: \"kubernetes.io/projected/5e1c80a4-7dd5-4341-b7a6-b534d026b54c-kube-api-access-g7528\") pod \"5e1c80a4-7dd5-4341-b7a6-b534d026b54c\" (UID: \"5e1c80a4-7dd5-4341-b7a6-b534d026b54c\") "
Nov 03 09:53:46.884996 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:46.884899    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e1c80a4-7dd5-4341-b7a6-b534d026b54c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e1c80a4-7dd5-4341-b7a6-b534d026b54c-maas-api-openshift-ca\") pod \"5e1c80a4-7dd5-4341-b7a6-b534d026b54c\" (UID: \"5e1c80a4-7dd5-4341-b7a6-b534d026b54c\") "
Nov 03 09:53:46.885382 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:46.885357    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e1c80a4-7dd5-4341-b7a6-b534d026b54c-maas-api-openshift-ca" pod "5e1c80a4-7dd5-4341-b7a6-b534d026b54c" (UID: "5e1c80a4-7dd5-4341-b7a6-b534d026b54c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:53:46.887015 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:46.886987    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e1c80a4-7dd5-4341-b7a6-b534d026b54c-maas-api-xks-ca" pod "5e1c80a4-7dd5-4341-b7a6-b534d026b54c" (UID: "5e1c80a4-7dd5-4341-b7a6-b534d026b54c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:53:46.887179 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:46.887161    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e1c80a4-7dd5-4341-b7a6-b534d026b54c-kube-api-access-g7528" pod "5e1c80a4-7dd5-4341-b7a6-b534d026b54c" (UID: "5e1c80a4-7dd5-4341-b7a6-b534d026b54c"). InnerVolumeSpecName "kube-api-access-g7528". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:53:46.938560 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:53:46.938475    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:53:46.986703 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:46.986633    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e1c80a4-7dd5-4341-b7a6-b534d026b54c-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:53:46.986703 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:46.986659    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7528\" (UniqueName: \"kubernetes.io/projected/5e1c80a4-7dd5-4341-b7a6-b534d026b54c-kube-api-access-g7528\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:53:46.986703 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:46.986673    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e1c80a4-7dd5-4341-b7a6-b534d026b54c-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:53:47.644448 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:47.644415    4473 generic.go:356] "Generic (PLEG): container finished" podID="3e87c013-ff41-4774-8e96-838df05ecc08" containerID="92f683f583b52c9933e2a325bbfd42b183218fe711e8a5f758e91ae962c110e7" exitCode=0
Nov 03 09:53:47.644872 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:47.644501    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df73bc75-zl9lt" event={"ID":"3e87c013-ff41-4774-8e96-838df05ecc08","Type":"ContainerDied","Data":"92f683f583b52c9933e2a325bbfd42b183218fe711e8a5f758e91ae962c110e7"}
Nov 03 09:53:47.645993 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:47.645969    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8d638a-zkz5t" event={"ID":"5e1c80a4-7dd5-4341-b7a6-b534d026b54c","Type":"ContainerDied","Data":"f0b50ad7ed488565385b189540c55bc7429d37163a2fabfd498199fb3a9fbb19"}
Nov 03 09:53:47.645993 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:47.645981    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d8d638a-zkz5t"
Nov 03 09:53:47.646194 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:47.646006    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0b50ad7ed488565385b189540c55bc7429d37163a2fabfd498199fb3a9fbb19"
Nov 03 09:53:48.799585 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:48.799554    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df73bc75-zl9lt"
Nov 03 09:53:48.909169 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:48.909076    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e87c013-ff41-4774-8e96-838df05ecc08-kube-api-access-gm8bm\" (UniqueName: \"kubernetes.io/projected/3e87c013-ff41-4774-8e96-838df05ecc08-kube-api-access-gm8bm\") pod \"3e87c013-ff41-4774-8e96-838df05ecc08\" (UID: \"3e87c013-ff41-4774-8e96-838df05ecc08\") "
Nov 03 09:53:48.909169 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:48.909123    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e87c013-ff41-4774-8e96-838df05ecc08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e87c013-ff41-4774-8e96-838df05ecc08-maas-api-openshift-ca\") pod \"3e87c013-ff41-4774-8e96-838df05ecc08\" (UID: \"3e87c013-ff41-4774-8e96-838df05ecc08\") "
Nov 03 09:53:48.909380 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:48.909190    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e87c013-ff41-4774-8e96-838df05ecc08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e87c013-ff41-4774-8e96-838df05ecc08-maas-api-xks-ca\") pod \"3e87c013-ff41-4774-8e96-838df05ecc08\" (UID: \"3e87c013-ff41-4774-8e96-838df05ecc08\") "
Nov 03 09:53:48.909501 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:48.909477    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e87c013-ff41-4774-8e96-838df05ecc08-maas-api-openshift-ca" pod "3e87c013-ff41-4774-8e96-838df05ecc08" (UID: "3e87c013-ff41-4774-8e96-838df05ecc08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:53:48.909598 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:48.909581    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e87c013-ff41-4774-8e96-838df05ecc08-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:53:48.912004 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:48.911972    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e87c013-ff41-4774-8e96-838df05ecc08-maas-api-xks-ca" pod "3e87c013-ff41-4774-8e96-838df05ecc08" (UID: "3e87c013-ff41-4774-8e96-838df05ecc08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:53:48.912143 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:48.912084    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e87c013-ff41-4774-8e96-838df05ecc08-kube-api-access-gm8bm" pod "3e87c013-ff41-4774-8e96-838df05ecc08" (UID: "3e87c013-ff41-4774-8e96-838df05ecc08"). InnerVolumeSpecName "kube-api-access-gm8bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:53:49.010927 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:49.010891    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm8bm\" (UniqueName: \"kubernetes.io/projected/3e87c013-ff41-4774-8e96-838df05ecc08-kube-api-access-gm8bm\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:53:49.010927 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:49.010929    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e87c013-ff41-4774-8e96-838df05ecc08-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:53:49.260800 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:49.260714    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f8439f5b-64dd678df-22vnq" podUID="80516551-d2d3-4bb6-897a-6da41d71e5c8" containerName="payload-processing" containerID="cri-o://96311e5b248037420185a91ca922914253b660946276e8d1f0a1665e85af1ab3" gracePeriod=25
Nov 03 09:53:49.374233 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:49.374179    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8439f5b-58b76f4b8c-x2dl8" podUID="159089a9-fca1-4bf3-ba81-feab46584505" containerName="payload-pre-processing" containerID="cri-o://13de1eb7a83755a554dcb25aaec8d58854d09ce49300921e284d6d48588fb205" gracePeriod=25
Nov 03 09:53:49.508024 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:49.507999    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f8439f5b-64dd678df-22vnq"
Nov 03 09:53:49.615935 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:49.615913    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80516551-d2d3-4bb6-897a-6da41d71e5c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80516551-d2d3-4bb6-897a-6da41d71e5c8-plugins-config-volume\") pod \"80516551-d2d3-4bb6-897a-6da41d71e5c8\" (UID: \"80516551-d2d3-4bb6-897a-6da41d71e5c8\") "
Nov 03 09:53:49.616061 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:49.615975    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80516551-d2d3-4bb6-897a-6da41d71e5c8-kube-api-access-gd2pk\" (UniqueName: \"kubernetes.io/projected/80516551-d2d3-4bb6-897a-6da41d71e5c8-kube-api-access-gd2pk\") pod \"80516551-d2d3-4bb6-897a-6da41d71e5c8\" (UID: \"80516551-d2d3-4bb6-897a-6da41d71e5c8\") "
Nov 03 09:53:49.616352 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:49.616327    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80516551-d2d3-4bb6-897a-6da41d71e5c8-plugins-config-volume" pod "80516551-d2d3-4bb6-897a-6da41d71e5c8" (UID: "80516551-d2d3-4bb6-897a-6da41d71e5c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:53:49.618276 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:49.618245    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80516551-d2d3-4bb6-897a-6da41d71e5c8-kube-api-access-gd2pk" pod "80516551-d2d3-4bb6-897a-6da41d71e5c8" (UID: "80516551-d2d3-4bb6-897a-6da41d71e5c8"). InnerVolumeSpecName "kube-api-access-gd2pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:53:49.621507 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:49.621487    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8439f5b-58b76f4b8c-x2dl8"
Nov 03 09:53:49.659807 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:49.659775    4473 generic.go:356] "Generic (PLEG): container finished" podID="159089a9-fca1-4bf3-ba81-feab46584505" containerID="13de1eb7a83755a554dcb25aaec8d58854d09ce49300921e284d6d48588fb205" exitCode=0
Nov 03 09:53:49.659964 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:49.659842    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8439f5b-58b76f4b8c-x2dl8"
Nov 03 09:53:49.659964 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:49.659875    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8439f5b-58b76f4b8c-x2dl8" event={"ID":"159089a9-fca1-4bf3-ba81-feab46584505","Type":"ContainerDied","Data":"13de1eb7a83755a554dcb25aaec8d58854d09ce49300921e284d6d48588fb205"}
Nov 03 09:53:49.659964 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:49.659928    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f8439f5b-58b76f4b8c-x2dl8" event={"ID":"159089a9-fca1-4bf3-ba81-feab46584505","Type":"ContainerDied","Data":"5d6cfd345a1cb123e266ab5fa1c8610f601483ef97d87b7e2553bae27ebe422f"}
Nov 03 09:53:49.660164 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:49.659976    4473 scope.go:122] "RemoveContainer" containerID="13de1eb7a83755a554dcb25aaec8d58854d09ce49300921e284d6d48588fb205"
Nov 03 09:53:49.661580 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:49.661556    4473 generic.go:356] "Generic (PLEG): container finished" podID="80516551-d2d3-4bb6-897a-6da41d71e5c8" containerID="96311e5b248037420185a91ca922914253b660946276e8d1f0a1665e85af1ab3" exitCode=0
Nov 03 09:53:49.661694 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:49.661595    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f8439f5b-64dd678df-22vnq"
Nov 03 09:53:49.661768 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:49.661696    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f8439f5b-64dd678df-22vnq" event={"ID":"80516551-d2d3-4bb6-897a-6da41d71e5c8","Type":"ContainerDied","Data":"96311e5b248037420185a91ca922914253b660946276e8d1f0a1665e85af1ab3"}
Nov 03 09:53:49.661768 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:49.661743    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f8439f5b-64dd678df-22vnq" event={"ID":"80516551-d2d3-4bb6-897a-6da41d71e5c8","Type":"ContainerDied","Data":"3a14bf1eea9ee031de6e7161e3ae02f433cb2888c6df35661f1f34efe58dcfed"}
Nov 03 09:53:49.663264 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:49.663244    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df73bc75-zl9lt"
Nov 03 09:53:49.663264 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:49.663259    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df73bc75-zl9lt" event={"ID":"3e87c013-ff41-4774-8e96-838df05ecc08","Type":"ContainerDied","Data":"62f0ac22e85cb9498f452952380722f2314e2630cac9dcd1ad148f6146efe232"}
Nov 03 09:53:49.663428 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:49.663293    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62f0ac22e85cb9498f452952380722f2314e2630cac9dcd1ad148f6146efe232"
Nov 03 09:53:49.670078 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:49.670061    4473 scope.go:122] "RemoveContainer" containerID="13de1eb7a83755a554dcb25aaec8d58854d09ce49300921e284d6d48588fb205"
Nov 03 09:53:49.670366 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:53:49.670346    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13de1eb7a83755a554dcb25aaec8d58854d09ce49300921e284d6d48588fb205\": container with ID starting with 13de1eb7a83755a554dcb25aaec8d58854d09ce49300921e284d6d48588fb205 not found: ID does not exist" containerID="13de1eb7a83755a554dcb25aaec8d58854d09ce49300921e284d6d48588fb205"
Nov 03 09:53:49.670415 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:49.670372    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13de1eb7a83755a554dcb25aaec8d58854d09ce49300921e284d6d48588fb205"} err="failed to get container status \"13de1eb7a83755a554dcb25aaec8d58854d09ce49300921e284d6d48588fb205\": rpc error: code = NotFound desc = could not find container \"13de1eb7a83755a554dcb25aaec8d58854d09ce49300921e284d6d48588fb205\": container with ID starting with 13de1eb7a83755a554dcb25aaec8d58854d09ce49300921e284d6d48588fb205 not found: ID does not exist"
Nov 03 09:53:49.670415 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:49.670389    4473 scope.go:122] "RemoveContainer" containerID="96311e5b248037420185a91ca922914253b660946276e8d1f0a1665e85af1ab3"
Nov 03 09:53:49.679311 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:49.679296    4473 scope.go:122] "RemoveContainer" containerID="96311e5b248037420185a91ca922914253b660946276e8d1f0a1665e85af1ab3"
Nov 03 09:53:49.679608 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:53:49.679583    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96311e5b248037420185a91ca922914253b660946276e8d1f0a1665e85af1ab3\": container with ID starting with 96311e5b248037420185a91ca922914253b660946276e8d1f0a1665e85af1ab3 not found: ID does not exist" containerID="96311e5b248037420185a91ca922914253b660946276e8d1f0a1665e85af1ab3"
Nov 03 09:53:49.679701 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:49.679611    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96311e5b248037420185a91ca922914253b660946276e8d1f0a1665e85af1ab3"} err="failed to get container status \"96311e5b248037420185a91ca922914253b660946276e8d1f0a1665e85af1ab3\": rpc error: code = NotFound desc = could not find container \"96311e5b248037420185a91ca922914253b660946276e8d1f0a1665e85af1ab3\": container with ID starting with 96311e5b248037420185a91ca922914253b660946276e8d1f0a1665e85af1ab3 not found: ID does not exist"
Nov 03 09:53:49.717135 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:49.717108    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/159089a9-fca1-4bf3-ba81-feab46584505-kube-api-access-4pgd7\" (UniqueName: \"kubernetes.io/projected/159089a9-fca1-4bf3-ba81-feab46584505-kube-api-access-4pgd7\") pod \"159089a9-fca1-4bf3-ba81-feab46584505\" (UID: \"159089a9-fca1-4bf3-ba81-feab46584505\") "
Nov 03 09:53:49.717259 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:49.717247    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/159089a9-fca1-4bf3-ba81-feab46584505-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/159089a9-fca1-4bf3-ba81-feab46584505-plugins-config-volume\") pod \"159089a9-fca1-4bf3-ba81-feab46584505\" (UID: \"159089a9-fca1-4bf3-ba81-feab46584505\") "
Nov 03 09:53:49.717490 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:49.717476    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80516551-d2d3-4bb6-897a-6da41d71e5c8-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:53:49.717490 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:49.717489    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd2pk\" (UniqueName: \"kubernetes.io/projected/80516551-d2d3-4bb6-897a-6da41d71e5c8-kube-api-access-gd2pk\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:53:49.717583 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:49.717566    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/159089a9-fca1-4bf3-ba81-feab46584505-plugins-config-volume" pod "159089a9-fca1-4bf3-ba81-feab46584505" (UID: "159089a9-fca1-4bf3-ba81-feab46584505"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:53:49.719218 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:49.719196    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/159089a9-fca1-4bf3-ba81-feab46584505-kube-api-access-4pgd7" pod "159089a9-fca1-4bf3-ba81-feab46584505" (UID: "159089a9-fca1-4bf3-ba81-feab46584505"). InnerVolumeSpecName "kube-api-access-4pgd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:53:49.819126 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:49.819059    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/159089a9-fca1-4bf3-ba81-feab46584505-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:53:49.819126 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:49.819087    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pgd7\" (UniqueName: \"kubernetes.io/projected/159089a9-fca1-4bf3-ba81-feab46584505-kube-api-access-4pgd7\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:53:49.858665 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:49.858635    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f8439f5b-64dd678df-22vnq"]
Nov 03 09:53:49.871326 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:49.871305    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f8439f5b-64dd678df-22vnq"]
Nov 03 09:53:50.085187 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:50.085107    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f8439f5b-58b76f4b8c-x2dl8"]
Nov 03 09:53:50.089135 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:50.089111    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f8439f5b-58b76f4b8c-x2dl8"]
Nov 03 09:53:51.869526 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:51.869495    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="159089a9-fca1-4bf3-ba81-feab46584505" path="/var/lib/kubelet/pods/159089a9-fca1-4bf3-ba81-feab46584505/volumes"
Nov 03 09:53:51.869904 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:51.869890    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80516551-d2d3-4bb6-897a-6da41d71e5c8" path="/var/lib/kubelet/pods/80516551-d2d3-4bb6-897a-6da41d71e5c8/volumes"
Nov 03 09:53:56.154468 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:56.154436    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7123bf8-s4l9j"]
Nov 03 09:53:56.154928 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:56.154915    4473 cpu_manager.go:414] "containerMap: removing container" podUID="159089a9-fca1-4bf3-ba81-feab46584505" containerName="payload-pre-processing"
Nov 03 09:53:56.155000 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:56.154929    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="159089a9-fca1-4bf3-ba81-feab46584505" containerName="payload-pre-processing"
Nov 03 09:53:56.155000 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:56.154964    4473 cpu_manager.go:414] "containerMap: removing container" podUID="80516551-d2d3-4bb6-897a-6da41d71e5c8" containerName="payload-processing"
Nov 03 09:53:56.155000 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:56.154971    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80516551-d2d3-4bb6-897a-6da41d71e5c8" containerName="payload-processing"
Nov 03 09:53:56.155000 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:56.154992    4473 cpu_manager.go:414] "containerMap: removing container" podUID="3e87c013-ff41-4774-8e96-838df05ecc08" containerName="revoke-keys"
Nov 03 09:53:56.155000 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:56.154997    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e87c013-ff41-4774-8e96-838df05ecc08" containerName="revoke-keys"
Nov 03 09:53:56.155159 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:56.155014    4473 cpu_manager.go:414] "containerMap: removing container" podUID="5e1c80a4-7dd5-4341-b7a6-b534d026b54c" containerName="revoke-keys"
Nov 03 09:53:56.155159 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:56.155019    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e1c80a4-7dd5-4341-b7a6-b534d026b54c" containerName="revoke-keys"
Nov 03 09:53:56.155159 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:56.155096    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e1c80a4-7dd5-4341-b7a6-b534d026b54c" containerName="revoke-keys"
Nov 03 09:53:56.155159 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:56.155107    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="159089a9-fca1-4bf3-ba81-feab46584505" containerName="payload-pre-processing"
Nov 03 09:53:56.155159 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:56.155118    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="80516551-d2d3-4bb6-897a-6da41d71e5c8" containerName="payload-processing"
Nov 03 09:53:56.155159 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:56.155126    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e87c013-ff41-4774-8e96-838df05ecc08" containerName="revoke-keys"
Nov 03 09:53:56.159777 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:56.159759    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7123bf8-s4l9j"
Nov 03 09:53:56.222970 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:56.219198    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7123bf8-s4l9j"]
Nov 03 09:53:56.265089 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:56.265057    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:53:56.265089 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:56.265070    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:53:56.265354 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:56.265126    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:53:56.265354 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:56.265253    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:53:56.297985 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:56.288344    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prbf8\" (UniqueName: \"kubernetes.io/projected/b6893d9f-cd25-4ded-afa9-c3e226ea9f9a-kube-api-access-prbf8\") pod \"maas-api-revoke-sub-d7123bf8-s4l9j\" (UID: \"b6893d9f-cd25-4ded-afa9-c3e226ea9f9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7123bf8-s4l9j"
Nov 03 09:53:56.297985 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:56.288410    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6893d9f-cd25-4ded-afa9-c3e226ea9f9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7123bf8-s4l9j\" (UID: \"b6893d9f-cd25-4ded-afa9-c3e226ea9f9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7123bf8-s4l9j"
Nov 03 09:53:56.297985 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:56.288559    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6893d9f-cd25-4ded-afa9-c3e226ea9f9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7123bf8-s4l9j\" (UID: \"b6893d9f-cd25-4ded-afa9-c3e226ea9f9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7123bf8-s4l9j"
Nov 03 09:53:56.390014 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:56.389976    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prbf8\" (UniqueName: \"kubernetes.io/projected/b6893d9f-cd25-4ded-afa9-c3e226ea9f9a-kube-api-access-prbf8\") pod \"maas-api-revoke-sub-d7123bf8-s4l9j\" (UID: \"b6893d9f-cd25-4ded-afa9-c3e226ea9f9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7123bf8-s4l9j"
Nov 03 09:53:56.390205 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:56.390031    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6893d9f-cd25-4ded-afa9-c3e226ea9f9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7123bf8-s4l9j\" (UID: \"b6893d9f-cd25-4ded-afa9-c3e226ea9f9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7123bf8-s4l9j"
Nov 03 09:53:56.390205 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:56.390157    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6893d9f-cd25-4ded-afa9-c3e226ea9f9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7123bf8-s4l9j\" (UID: \"b6893d9f-cd25-4ded-afa9-c3e226ea9f9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7123bf8-s4l9j"
Nov 03 09:53:56.390806 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:56.390782    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6893d9f-cd25-4ded-afa9-c3e226ea9f9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7123bf8-s4l9j\" (UID: \"b6893d9f-cd25-4ded-afa9-c3e226ea9f9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7123bf8-s4l9j"
Nov 03 09:53:56.392670 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:56.392636    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6893d9f-cd25-4ded-afa9-c3e226ea9f9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7123bf8-s4l9j\" (UID: \"b6893d9f-cd25-4ded-afa9-c3e226ea9f9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7123bf8-s4l9j"
Nov 03 09:53:56.449424 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:56.449351    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prbf8\" (UniqueName: \"kubernetes.io/projected/b6893d9f-cd25-4ded-afa9-c3e226ea9f9a-kube-api-access-prbf8\") pod \"maas-api-revoke-sub-d7123bf8-s4l9j\" (UID: \"b6893d9f-cd25-4ded-afa9-c3e226ea9f9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7123bf8-s4l9j"
Nov 03 09:53:56.473676 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:56.473650    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7123bf8-s4l9j"
Nov 03 09:53:56.851935 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:56.851903    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7123bf8-s4l9j"]
Nov 03 09:53:56.853445 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:53:56.853411    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6893d9f_cd25_4ded_afa9_c3e226ea9f9a.slice/crio-67ee4dfe3427ee8036080d377121bd55818f8f69237dfe5b740e56d16f3194f3 WatchSource:0}: Error finding container 67ee4dfe3427ee8036080d377121bd55818f8f69237dfe5b740e56d16f3194f3: Status 404 returned error can't find the container with id 67ee4dfe3427ee8036080d377121bd55818f8f69237dfe5b740e56d16f3194f3
Nov 03 09:53:57.712696 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:57.712657    4473 generic.go:356] "Generic (PLEG): container finished" podID="b6893d9f-cd25-4ded-afa9-c3e226ea9f9a" containerID="59c404aa6d80b483b04967dfe75b8de0ede492fabd07f0d047d6f157dc0375a4" exitCode=0
Nov 03 09:53:57.713077 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:57.712711    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7123bf8-s4l9j" event={"ID":"b6893d9f-cd25-4ded-afa9-c3e226ea9f9a","Type":"ContainerDied","Data":"59c404aa6d80b483b04967dfe75b8de0ede492fabd07f0d047d6f157dc0375a4"}
Nov 03 09:53:57.713077 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:57.712782    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7123bf8-s4l9j" event={"ID":"b6893d9f-cd25-4ded-afa9-c3e226ea9f9a","Type":"ContainerStarted","Data":"67ee4dfe3427ee8036080d377121bd55818f8f69237dfe5b740e56d16f3194f3"}
Nov 03 09:53:58.869302 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:58.869280    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7123bf8-s4l9j"
Nov 03 09:53:58.917568 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:58.917539    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6893d9f-cd25-4ded-afa9-c3e226ea9f9a-kube-api-access-prbf8\" (UniqueName: \"kubernetes.io/projected/b6893d9f-cd25-4ded-afa9-c3e226ea9f9a-kube-api-access-prbf8\") pod \"b6893d9f-cd25-4ded-afa9-c3e226ea9f9a\" (UID: \"b6893d9f-cd25-4ded-afa9-c3e226ea9f9a\") "
Nov 03 09:53:58.917763 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:58.917743    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6893d9f-cd25-4ded-afa9-c3e226ea9f9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6893d9f-cd25-4ded-afa9-c3e226ea9f9a-maas-api-xks-ca\") pod \"b6893d9f-cd25-4ded-afa9-c3e226ea9f9a\" (UID: \"b6893d9f-cd25-4ded-afa9-c3e226ea9f9a\") "
Nov 03 09:53:58.917843 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:58.917827    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6893d9f-cd25-4ded-afa9-c3e226ea9f9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6893d9f-cd25-4ded-afa9-c3e226ea9f9a-maas-api-openshift-ca\") pod \"b6893d9f-cd25-4ded-afa9-c3e226ea9f9a\" (UID: \"b6893d9f-cd25-4ded-afa9-c3e226ea9f9a\") "
Nov 03 09:53:58.918127 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:58.918099    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6893d9f-cd25-4ded-afa9-c3e226ea9f9a-maas-api-openshift-ca" pod "b6893d9f-cd25-4ded-afa9-c3e226ea9f9a" (UID: "b6893d9f-cd25-4ded-afa9-c3e226ea9f9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:53:58.918400 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:58.918335    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6893d9f-cd25-4ded-afa9-c3e226ea9f9a-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:53:58.919922 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:58.919898    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6893d9f-cd25-4ded-afa9-c3e226ea9f9a-maas-api-xks-ca" pod "b6893d9f-cd25-4ded-afa9-c3e226ea9f9a" (UID: "b6893d9f-cd25-4ded-afa9-c3e226ea9f9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:53:58.920105 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:58.920085    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6893d9f-cd25-4ded-afa9-c3e226ea9f9a-kube-api-access-prbf8" pod "b6893d9f-cd25-4ded-afa9-c3e226ea9f9a" (UID: "b6893d9f-cd25-4ded-afa9-c3e226ea9f9a"). InnerVolumeSpecName "kube-api-access-prbf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:53:59.019994 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:59.019904    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prbf8\" (UniqueName: \"kubernetes.io/projected/b6893d9f-cd25-4ded-afa9-c3e226ea9f9a-kube-api-access-prbf8\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:53:59.019994 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:59.019939    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6893d9f-cd25-4ded-afa9-c3e226ea9f9a-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:53:59.725479 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:59.725449    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7123bf8-s4l9j"
Nov 03 09:53:59.725656 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:59.725480    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7123bf8-s4l9j" event={"ID":"b6893d9f-cd25-4ded-afa9-c3e226ea9f9a","Type":"ContainerDied","Data":"67ee4dfe3427ee8036080d377121bd55818f8f69237dfe5b740e56d16f3194f3"}
Nov 03 09:53:59.725656 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:53:59.725526    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67ee4dfe3427ee8036080d377121bd55818f8f69237dfe5b740e56d16f3194f3"
Nov 03 09:54:08.212073 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:08.212032    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:54:11.385271 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:11.385234    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2093da2-wdlxd"]
Nov 03 09:54:11.385763 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:11.385724    4473 cpu_manager.go:414] "containerMap: removing container" podUID="b6893d9f-cd25-4ded-afa9-c3e226ea9f9a" containerName="revoke-keys"
Nov 03 09:54:11.385763 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:11.385735    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6893d9f-cd25-4ded-afa9-c3e226ea9f9a" containerName="revoke-keys"
Nov 03 09:54:11.385842 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:11.385823    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6893d9f-cd25-4ded-afa9-c3e226ea9f9a" containerName="revoke-keys"
Nov 03 09:54:11.389346 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:11.389328    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2093da2-wdlxd"
Nov 03 09:54:11.447525 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:11.447492    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2093da2-wdlxd"]
Nov 03 09:54:11.469530 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:11.469501    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08906c75-24d6-4cdc-8220-f93ca8d85d7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2093da2-wdlxd\" (UID: \"08906c75-24d6-4cdc-8220-f93ca8d85d7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2093da2-wdlxd"
Nov 03 09:54:11.469703 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:11.469570    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wzz2\" (UniqueName: \"kubernetes.io/projected/08906c75-24d6-4cdc-8220-f93ca8d85d7a-kube-api-access-4wzz2\") pod \"maas-api-revoke-sub-d2093da2-wdlxd\" (UID: \"08906c75-24d6-4cdc-8220-f93ca8d85d7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2093da2-wdlxd"
Nov 03 09:54:11.469703 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:11.469660    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08906c75-24d6-4cdc-8220-f93ca8d85d7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2093da2-wdlxd\" (UID: \"08906c75-24d6-4cdc-8220-f93ca8d85d7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2093da2-wdlxd"
Nov 03 09:54:11.493813 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:11.493775    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:54:11.493813 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:11.493773    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:54:11.493813 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:11.493808    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:54:11.494136 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:11.493853    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:54:11.570693 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:11.570657    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08906c75-24d6-4cdc-8220-f93ca8d85d7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2093da2-wdlxd\" (UID: \"08906c75-24d6-4cdc-8220-f93ca8d85d7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2093da2-wdlxd"
Nov 03 09:54:11.570893 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:11.570709    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wzz2\" (UniqueName: \"kubernetes.io/projected/08906c75-24d6-4cdc-8220-f93ca8d85d7a-kube-api-access-4wzz2\") pod \"maas-api-revoke-sub-d2093da2-wdlxd\" (UID: \"08906c75-24d6-4cdc-8220-f93ca8d85d7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2093da2-wdlxd"
Nov 03 09:54:11.570893 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:11.570740    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08906c75-24d6-4cdc-8220-f93ca8d85d7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2093da2-wdlxd\" (UID: \"08906c75-24d6-4cdc-8220-f93ca8d85d7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2093da2-wdlxd"
Nov 03 09:54:11.571682 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:11.571653    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08906c75-24d6-4cdc-8220-f93ca8d85d7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2093da2-wdlxd\" (UID: \"08906c75-24d6-4cdc-8220-f93ca8d85d7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2093da2-wdlxd"
Nov 03 09:54:11.573975 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:11.573914    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08906c75-24d6-4cdc-8220-f93ca8d85d7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2093da2-wdlxd\" (UID: \"08906c75-24d6-4cdc-8220-f93ca8d85d7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2093da2-wdlxd"
Nov 03 09:54:11.629825 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:11.629789    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wzz2\" (UniqueName: \"kubernetes.io/projected/08906c75-24d6-4cdc-8220-f93ca8d85d7a-kube-api-access-4wzz2\") pod \"maas-api-revoke-sub-d2093da2-wdlxd\" (UID: \"08906c75-24d6-4cdc-8220-f93ca8d85d7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2093da2-wdlxd"
Nov 03 09:54:11.703335 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:11.703255    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2093da2-wdlxd"
Nov 03 09:54:12.086042 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:12.086014    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2093da2-wdlxd"]
Nov 03 09:54:12.086649 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:54:12.086626    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08906c75_24d6_4cdc_8220_f93ca8d85d7a.slice/crio-6e3e85127f198c837b55e9f214c763093e16c2e9a601c50ac4977737a81ce9d5 WatchSource:0}: Error finding container 6e3e85127f198c837b55e9f214c763093e16c2e9a601c50ac4977737a81ce9d5: Status 404 returned error can't find the container with id 6e3e85127f198c837b55e9f214c763093e16c2e9a601c50ac4977737a81ce9d5
Nov 03 09:54:12.810114 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:12.810073    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2093da2-wdlxd" event={"ID":"08906c75-24d6-4cdc-8220-f93ca8d85d7a","Type":"ContainerStarted","Data":"d7054a28bca74f4cffb3f81bb236acd7e809353af3816e823bdd8122c8344bb9"}
Nov 03 09:54:12.810528 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:12.810131    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2093da2-wdlxd" event={"ID":"08906c75-24d6-4cdc-8220-f93ca8d85d7a","Type":"ContainerStarted","Data":"6e3e85127f198c837b55e9f214c763093e16c2e9a601c50ac4977737a81ce9d5"}
Nov 03 09:54:12.927147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:12.927101    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2093da2-wdlxd" podStartSLOduration=1.9270863440000001 podStartE2EDuration="1.927086344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:54:12.92552552 +0000 UTC m=+4477.194588956" watchObservedRunningTime="2026-11-03 09:54:12.927086344 +0000 UTC m=+4477.196149782"
Nov 03 09:54:13.817552 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:13.817521    4473 generic.go:356] "Generic (PLEG): container finished" podID="08906c75-24d6-4cdc-8220-f93ca8d85d7a" containerID="d7054a28bca74f4cffb3f81bb236acd7e809353af3816e823bdd8122c8344bb9" exitCode=0
Nov 03 09:54:13.817923 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:13.817602    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2093da2-wdlxd" event={"ID":"08906c75-24d6-4cdc-8220-f93ca8d85d7a","Type":"ContainerDied","Data":"d7054a28bca74f4cffb3f81bb236acd7e809353af3816e823bdd8122c8344bb9"}
Nov 03 09:54:14.965958 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:14.965919    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2093da2-wdlxd"
Nov 03 09:54:15.109685 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:15.109616    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08906c75-24d6-4cdc-8220-f93ca8d85d7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08906c75-24d6-4cdc-8220-f93ca8d85d7a-maas-api-xks-ca\") pod \"08906c75-24d6-4cdc-8220-f93ca8d85d7a\" (UID: \"08906c75-24d6-4cdc-8220-f93ca8d85d7a\") "
Nov 03 09:54:15.109829 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:15.109742    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08906c75-24d6-4cdc-8220-f93ca8d85d7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08906c75-24d6-4cdc-8220-f93ca8d85d7a-maas-api-openshift-ca\") pod \"08906c75-24d6-4cdc-8220-f93ca8d85d7a\" (UID: \"08906c75-24d6-4cdc-8220-f93ca8d85d7a\") "
Nov 03 09:54:15.109875 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:15.109849    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08906c75-24d6-4cdc-8220-f93ca8d85d7a-kube-api-access-4wzz2\" (UniqueName: \"kubernetes.io/projected/08906c75-24d6-4cdc-8220-f93ca8d85d7a-kube-api-access-4wzz2\") pod \"08906c75-24d6-4cdc-8220-f93ca8d85d7a\" (UID: \"08906c75-24d6-4cdc-8220-f93ca8d85d7a\") "
Nov 03 09:54:15.110100 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:15.110075    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08906c75-24d6-4cdc-8220-f93ca8d85d7a-maas-api-openshift-ca" pod "08906c75-24d6-4cdc-8220-f93ca8d85d7a" (UID: "08906c75-24d6-4cdc-8220-f93ca8d85d7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:54:15.110276 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:15.110262    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08906c75-24d6-4cdc-8220-f93ca8d85d7a-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:15.111976 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:15.111932    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08906c75-24d6-4cdc-8220-f93ca8d85d7a-maas-api-xks-ca" pod "08906c75-24d6-4cdc-8220-f93ca8d85d7a" (UID: "08906c75-24d6-4cdc-8220-f93ca8d85d7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:54:15.112093 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:15.112075    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08906c75-24d6-4cdc-8220-f93ca8d85d7a-kube-api-access-4wzz2" pod "08906c75-24d6-4cdc-8220-f93ca8d85d7a" (UID: "08906c75-24d6-4cdc-8220-f93ca8d85d7a"). InnerVolumeSpecName "kube-api-access-4wzz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:54:15.211700 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:15.211670    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wzz2\" (UniqueName: \"kubernetes.io/projected/08906c75-24d6-4cdc-8220-f93ca8d85d7a-kube-api-access-4wzz2\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:15.211700 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:15.211697    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08906c75-24d6-4cdc-8220-f93ca8d85d7a-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:15.505701 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:15.505666    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g"]
Nov 03 09:54:15.511295 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:15.511267    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5"]
Nov 03 09:54:15.556058 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:15.556024    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm"]
Nov 03 09:54:15.559068 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:15.559039    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh"]
Nov 03 09:54:15.832014 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:15.831918    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2093da2-wdlxd"
Nov 03 09:54:15.832014 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:15.831992    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2093da2-wdlxd" event={"ID":"08906c75-24d6-4cdc-8220-f93ca8d85d7a","Type":"ContainerDied","Data":"6e3e85127f198c837b55e9f214c763093e16c2e9a601c50ac4977737a81ce9d5"}
Nov 03 09:54:15.832215 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:15.832069    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e3e85127f198c837b55e9f214c763093e16c2e9a601c50ac4977737a81ce9d5"
Nov 03 09:54:16.939019 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:54:16.938982    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:54:21.568528 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:21.568494    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-750cbe2e-kgmtv"]
Nov 03 09:54:21.569044 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:21.569028    4473 cpu_manager.go:414] "containerMap: removing container" podUID="08906c75-24d6-4cdc-8220-f93ca8d85d7a" containerName="revoke-keys"
Nov 03 09:54:21.569102 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:21.569046    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08906c75-24d6-4cdc-8220-f93ca8d85d7a" containerName="revoke-keys"
Nov 03 09:54:21.569179 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:21.569168    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="08906c75-24d6-4cdc-8220-f93ca8d85d7a" containerName="revoke-keys"
Nov 03 09:54:21.572181 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:21.572166    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-750cbe2e-kgmtv"
Nov 03 09:54:21.629586 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:21.629558    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-750cbe2e-kgmtv"]
Nov 03 09:54:21.676238 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:21.676212    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:54:21.676392 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:21.676269    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:54:21.676392 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:21.676353    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:54:21.676392 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:21.676377    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:54:21.683002 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:21.682982    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73981a22-201c-42d3-b1d6-81afe3c9d24f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-750cbe2e-kgmtv\" (UID: \"73981a22-201c-42d3-b1d6-81afe3c9d24f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-750cbe2e-kgmtv"
Nov 03 09:54:21.683090 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:21.683010    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73981a22-201c-42d3-b1d6-81afe3c9d24f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-750cbe2e-kgmtv\" (UID: \"73981a22-201c-42d3-b1d6-81afe3c9d24f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-750cbe2e-kgmtv"
Nov 03 09:54:21.683186 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:21.683166    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlc79\" (UniqueName: \"kubernetes.io/projected/73981a22-201c-42d3-b1d6-81afe3c9d24f-kube-api-access-wlc79\") pod \"maas-api-revoke-sub-750cbe2e-kgmtv\" (UID: \"73981a22-201c-42d3-b1d6-81afe3c9d24f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-750cbe2e-kgmtv"
Nov 03 09:54:21.784223 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:21.784198    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlc79\" (UniqueName: \"kubernetes.io/projected/73981a22-201c-42d3-b1d6-81afe3c9d24f-kube-api-access-wlc79\") pod \"maas-api-revoke-sub-750cbe2e-kgmtv\" (UID: \"73981a22-201c-42d3-b1d6-81afe3c9d24f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-750cbe2e-kgmtv"
Nov 03 09:54:21.784374 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:21.784255    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73981a22-201c-42d3-b1d6-81afe3c9d24f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-750cbe2e-kgmtv\" (UID: \"73981a22-201c-42d3-b1d6-81afe3c9d24f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-750cbe2e-kgmtv"
Nov 03 09:54:21.784374 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:21.784283    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73981a22-201c-42d3-b1d6-81afe3c9d24f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-750cbe2e-kgmtv\" (UID: \"73981a22-201c-42d3-b1d6-81afe3c9d24f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-750cbe2e-kgmtv"
Nov 03 09:54:21.785032 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:21.785009    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73981a22-201c-42d3-b1d6-81afe3c9d24f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-750cbe2e-kgmtv\" (UID: \"73981a22-201c-42d3-b1d6-81afe3c9d24f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-750cbe2e-kgmtv"
Nov 03 09:54:21.786733 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:21.786711    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73981a22-201c-42d3-b1d6-81afe3c9d24f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-750cbe2e-kgmtv\" (UID: \"73981a22-201c-42d3-b1d6-81afe3c9d24f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-750cbe2e-kgmtv"
Nov 03 09:54:21.843203 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:21.843146    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlc79\" (UniqueName: \"kubernetes.io/projected/73981a22-201c-42d3-b1d6-81afe3c9d24f-kube-api-access-wlc79\") pod \"maas-api-revoke-sub-750cbe2e-kgmtv\" (UID: \"73981a22-201c-42d3-b1d6-81afe3c9d24f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-750cbe2e-kgmtv"
Nov 03 09:54:21.885300 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:21.885274    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-750cbe2e-kgmtv"
Nov 03 09:54:22.283722 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:22.283689    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-750cbe2e-kgmtv"]
Nov 03 09:54:22.284884 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:54:22.284859    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73981a22_201c_42d3_b1d6_81afe3c9d24f.slice/crio-7468b406c173fdf569a5a40f2124a3ba5a6de59e004f31ae9481d541a7413c27 WatchSource:0}: Error finding container 7468b406c173fdf569a5a40f2124a3ba5a6de59e004f31ae9481d541a7413c27: Status 404 returned error can't find the container with id 7468b406c173fdf569a5a40f2124a3ba5a6de59e004f31ae9481d541a7413c27
Nov 03 09:54:22.878882 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:22.878841    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-750cbe2e-kgmtv" event={"ID":"73981a22-201c-42d3-b1d6-81afe3c9d24f","Type":"ContainerStarted","Data":"03af178a474f1198f9eb70fc2b6c9d901e182b867b59547f1749245c95933263"}
Nov 03 09:54:22.879300 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:22.878904    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-750cbe2e-kgmtv" event={"ID":"73981a22-201c-42d3-b1d6-81afe3c9d24f","Type":"ContainerStarted","Data":"7468b406c173fdf569a5a40f2124a3ba5a6de59e004f31ae9481d541a7413c27"}
Nov 03 09:54:22.996616 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:22.996557    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-750cbe2e-kgmtv" podStartSLOduration=1.996542284 podStartE2EDuration="1.996542284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:54:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:54:22.994203062 +0000 UTC m=+4487.263266496" watchObservedRunningTime="2026-11-03 09:54:22.996542284 +0000 UTC m=+4487.265605718"
Nov 03 09:54:23.648479 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:23.648448    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07d668f1-k49m4"]
Nov 03 09:54:23.652186 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:23.652169    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d668f1-k49m4"
Nov 03 09:54:23.704342 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:23.704312    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89f277d9-a48e-4fbe-b866-b26ef10468d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07d668f1-k49m4\" (UID: \"89f277d9-a48e-4fbe-b866-b26ef10468d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d668f1-k49m4"
Nov 03 09:54:23.704450 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:23.704353    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89f277d9-a48e-4fbe-b866-b26ef10468d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07d668f1-k49m4\" (UID: \"89f277d9-a48e-4fbe-b866-b26ef10468d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d668f1-k49m4"
Nov 03 09:54:23.704497 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:23.704445    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpddx\" (UniqueName: \"kubernetes.io/projected/89f277d9-a48e-4fbe-b866-b26ef10468d4-kube-api-access-xpddx\") pod \"maas-api-revoke-sub-07d668f1-k49m4\" (UID: \"89f277d9-a48e-4fbe-b866-b26ef10468d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d668f1-k49m4"
Nov 03 09:54:23.709643 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:23.709608    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07d668f1-k49m4"]
Nov 03 09:54:23.805506 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:23.805472    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89f277d9-a48e-4fbe-b866-b26ef10468d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07d668f1-k49m4\" (UID: \"89f277d9-a48e-4fbe-b866-b26ef10468d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d668f1-k49m4"
Nov 03 09:54:23.805684 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:23.805515    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89f277d9-a48e-4fbe-b866-b26ef10468d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07d668f1-k49m4\" (UID: \"89f277d9-a48e-4fbe-b866-b26ef10468d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d668f1-k49m4"
Nov 03 09:54:23.805684 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:23.805545    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpddx\" (UniqueName: \"kubernetes.io/projected/89f277d9-a48e-4fbe-b866-b26ef10468d4-kube-api-access-xpddx\") pod \"maas-api-revoke-sub-07d668f1-k49m4\" (UID: \"89f277d9-a48e-4fbe-b866-b26ef10468d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d668f1-k49m4"
Nov 03 09:54:23.806269 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:23.806243    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89f277d9-a48e-4fbe-b866-b26ef10468d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07d668f1-k49m4\" (UID: \"89f277d9-a48e-4fbe-b866-b26ef10468d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d668f1-k49m4"
Nov 03 09:54:23.808033 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:23.808015    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89f277d9-a48e-4fbe-b866-b26ef10468d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07d668f1-k49m4\" (UID: \"89f277d9-a48e-4fbe-b866-b26ef10468d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d668f1-k49m4"
Nov 03 09:54:23.864868 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:23.864838    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpddx\" (UniqueName: \"kubernetes.io/projected/89f277d9-a48e-4fbe-b866-b26ef10468d4-kube-api-access-xpddx\") pod \"maas-api-revoke-sub-07d668f1-k49m4\" (UID: \"89f277d9-a48e-4fbe-b866-b26ef10468d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d668f1-k49m4"
Nov 03 09:54:23.886468 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:23.886441    4473 generic.go:356] "Generic (PLEG): container finished" podID="73981a22-201c-42d3-b1d6-81afe3c9d24f" containerID="03af178a474f1198f9eb70fc2b6c9d901e182b867b59547f1749245c95933263" exitCode=0
Nov 03 09:54:23.886823 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:23.886472    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-750cbe2e-kgmtv" event={"ID":"73981a22-201c-42d3-b1d6-81afe3c9d24f","Type":"ContainerDied","Data":"03af178a474f1198f9eb70fc2b6c9d901e182b867b59547f1749245c95933263"}
Nov 03 09:54:23.965215 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:23.965154    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d668f1-k49m4"
Nov 03 09:54:24.344864 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:24.344840    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07d668f1-k49m4"]
Nov 03 09:54:24.345908 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:54:24.345880    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89f277d9_a48e_4fbe_b866_b26ef10468d4.slice/crio-cac5a6e86a6e3eb03b495bba910089bf4daea479dae0d8db37721ba0f12e356f WatchSource:0}: Error finding container cac5a6e86a6e3eb03b495bba910089bf4daea479dae0d8db37721ba0f12e356f: Status 404 returned error can't find the container with id cac5a6e86a6e3eb03b495bba910089bf4daea479dae0d8db37721ba0f12e356f
Nov 03 09:54:24.894704 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:24.894610    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d668f1-k49m4" event={"ID":"89f277d9-a48e-4fbe-b866-b26ef10468d4","Type":"ContainerStarted","Data":"b54b6a7e98fc2cda172f57494563ee392e2c7a4a548964d7974fb0afb6039986"}
Nov 03 09:54:24.894704 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:24.894668    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d668f1-k49m4" event={"ID":"89f277d9-a48e-4fbe-b866-b26ef10468d4","Type":"ContainerStarted","Data":"cac5a6e86a6e3eb03b495bba910089bf4daea479dae0d8db37721ba0f12e356f"}
Nov 03 09:54:25.013348 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:25.013280    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d668f1-k49m4" podStartSLOduration=2.013262798 podStartE2EDuration="2.013262798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:54:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:54:25.009002061 +0000 UTC m=+4489.278065512" watchObservedRunningTime="2026-11-03 09:54:25.013262798 +0000 UTC m=+4489.282326234"
Nov 03 09:54:25.039333 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:25.039307    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-750cbe2e-kgmtv"
Nov 03 09:54:25.119224 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:25.119196    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73981a22-201c-42d3-b1d6-81afe3c9d24f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73981a22-201c-42d3-b1d6-81afe3c9d24f-maas-api-openshift-ca\") pod \"73981a22-201c-42d3-b1d6-81afe3c9d24f\" (UID: \"73981a22-201c-42d3-b1d6-81afe3c9d24f\") "
Nov 03 09:54:25.119390 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:25.119259    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73981a22-201c-42d3-b1d6-81afe3c9d24f-kube-api-access-wlc79\" (UniqueName: \"kubernetes.io/projected/73981a22-201c-42d3-b1d6-81afe3c9d24f-kube-api-access-wlc79\") pod \"73981a22-201c-42d3-b1d6-81afe3c9d24f\" (UID: \"73981a22-201c-42d3-b1d6-81afe3c9d24f\") "
Nov 03 09:54:25.119390 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:25.119357    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73981a22-201c-42d3-b1d6-81afe3c9d24f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73981a22-201c-42d3-b1d6-81afe3c9d24f-maas-api-xks-ca\") pod \"73981a22-201c-42d3-b1d6-81afe3c9d24f\" (UID: \"73981a22-201c-42d3-b1d6-81afe3c9d24f\") "
Nov 03 09:54:25.119630 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:25.119607    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73981a22-201c-42d3-b1d6-81afe3c9d24f-maas-api-openshift-ca" pod "73981a22-201c-42d3-b1d6-81afe3c9d24f" (UID: "73981a22-201c-42d3-b1d6-81afe3c9d24f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:54:25.119803 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:25.119781    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73981a22-201c-42d3-b1d6-81afe3c9d24f-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:25.121614 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:25.121585    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73981a22-201c-42d3-b1d6-81afe3c9d24f-kube-api-access-wlc79" pod "73981a22-201c-42d3-b1d6-81afe3c9d24f" (UID: "73981a22-201c-42d3-b1d6-81afe3c9d24f"). InnerVolumeSpecName "kube-api-access-wlc79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:54:25.121730 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:25.121672    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73981a22-201c-42d3-b1d6-81afe3c9d24f-maas-api-xks-ca" pod "73981a22-201c-42d3-b1d6-81afe3c9d24f" (UID: "73981a22-201c-42d3-b1d6-81afe3c9d24f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:54:25.221356 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:25.221327    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlc79\" (UniqueName: \"kubernetes.io/projected/73981a22-201c-42d3-b1d6-81afe3c9d24f-kube-api-access-wlc79\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:25.221356 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:25.221358    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73981a22-201c-42d3-b1d6-81afe3c9d24f-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:25.902351 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:25.902321    4473 generic.go:356] "Generic (PLEG): container finished" podID="89f277d9-a48e-4fbe-b866-b26ef10468d4" containerID="b54b6a7e98fc2cda172f57494563ee392e2c7a4a548964d7974fb0afb6039986" exitCode=0
Nov 03 09:54:25.902730 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:25.902402    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d668f1-k49m4" event={"ID":"89f277d9-a48e-4fbe-b866-b26ef10468d4","Type":"ContainerDied","Data":"b54b6a7e98fc2cda172f57494563ee392e2c7a4a548964d7974fb0afb6039986"}
Nov 03 09:54:25.903901 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:25.903879    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-750cbe2e-kgmtv" event={"ID":"73981a22-201c-42d3-b1d6-81afe3c9d24f","Type":"ContainerDied","Data":"7468b406c173fdf569a5a40f2124a3ba5a6de59e004f31ae9481d541a7413c27"}
Nov 03 09:54:25.904035 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:25.903911    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-750cbe2e-kgmtv"
Nov 03 09:54:25.904035 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:25.903922    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7468b406c173fdf569a5a40f2124a3ba5a6de59e004f31ae9481d541a7413c27"
Nov 03 09:54:27.042159 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:27.042121    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d668f1-k49m4"
Nov 03 09:54:27.158287 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:27.158260    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89f277d9-a48e-4fbe-b866-b26ef10468d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89f277d9-a48e-4fbe-b866-b26ef10468d4-maas-api-xks-ca\") pod \"89f277d9-a48e-4fbe-b866-b26ef10468d4\" (UID: \"89f277d9-a48e-4fbe-b866-b26ef10468d4\") "
Nov 03 09:54:27.158902 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:27.158876    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89f277d9-a48e-4fbe-b866-b26ef10468d4-kube-api-access-xpddx\" (UniqueName: \"kubernetes.io/projected/89f277d9-a48e-4fbe-b866-b26ef10468d4-kube-api-access-xpddx\") pod \"89f277d9-a48e-4fbe-b866-b26ef10468d4\" (UID: \"89f277d9-a48e-4fbe-b866-b26ef10468d4\") "
Nov 03 09:54:27.159113 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:27.159086    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89f277d9-a48e-4fbe-b866-b26ef10468d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89f277d9-a48e-4fbe-b866-b26ef10468d4-maas-api-openshift-ca\") pod \"89f277d9-a48e-4fbe-b866-b26ef10468d4\" (UID: \"89f277d9-a48e-4fbe-b866-b26ef10468d4\") "
Nov 03 09:54:27.159588 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:27.159557    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89f277d9-a48e-4fbe-b866-b26ef10468d4-maas-api-openshift-ca" pod "89f277d9-a48e-4fbe-b866-b26ef10468d4" (UID: "89f277d9-a48e-4fbe-b866-b26ef10468d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:54:27.161114 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:27.161030    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89f277d9-a48e-4fbe-b866-b26ef10468d4-maas-api-xks-ca" pod "89f277d9-a48e-4fbe-b866-b26ef10468d4" (UID: "89f277d9-a48e-4fbe-b866-b26ef10468d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:54:27.163904 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:27.163883    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89f277d9-a48e-4fbe-b866-b26ef10468d4-kube-api-access-xpddx" pod "89f277d9-a48e-4fbe-b866-b26ef10468d4" (UID: "89f277d9-a48e-4fbe-b866-b26ef10468d4"). InnerVolumeSpecName "kube-api-access-xpddx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:54:27.260926 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:27.260896    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89f277d9-a48e-4fbe-b866-b26ef10468d4-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:27.260926 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:27.260921    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpddx\" (UniqueName: \"kubernetes.io/projected/89f277d9-a48e-4fbe-b866-b26ef10468d4-kube-api-access-xpddx\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:27.260926 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:27.260932    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89f277d9-a48e-4fbe-b866-b26ef10468d4-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:27.927126 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:27.927092    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d668f1-k49m4"
Nov 03 09:54:27.927321 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:27.927122    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d668f1-k49m4" event={"ID":"89f277d9-a48e-4fbe-b866-b26ef10468d4","Type":"ContainerDied","Data":"cac5a6e86a6e3eb03b495bba910089bf4daea479dae0d8db37721ba0f12e356f"}
Nov 03 09:54:27.927321 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:27.927207    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cac5a6e86a6e3eb03b495bba910089bf4daea479dae0d8db37721ba0f12e356f"
Nov 03 09:54:30.527146 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.527100    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g" podUID="9e7f39b3-ce66-4b2d-abbc-8f078d527a2d" containerName="main" containerID="cri-o://6cbfa3600b6ec31caa148e0864bef52b0805a33bb415ff10a049c4154e11b8f8" gracePeriod=45
Nov 03 09:54:30.531088 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.531053    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5" podUID="2e42501e-d39d-4ae1-91c7-76c3c171538e" containerName="main" containerID="cri-o://804a116da52cdfcc4d7e8f31ff5aae521e8c0d7291a469edb1b72c1fa18a1a4c" gracePeriod=45
Nov 03 09:54:30.576958 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.576913    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm" podUID="60b1d7ef-16c6-4b6a-973b-4a251593eac7" containerName="main" containerID="cri-o://3744118d3b2020f3e6089947695f142df33af61398f13df40b08c08432f9f451" gracePeriod=45
Nov 03 09:54:30.578591 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.578564    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh" podUID="00bab831-c667-415d-b6b0-81fff4d57762" containerName="main" containerID="cri-o://3a357ddbfa22890aeeb1111586d18167a593964ad6d736077e6169c86223dbcf" gracePeriod=45
Nov 03 09:54:30.856736 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.856703    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g"
Nov 03 09:54:30.919575 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.919555    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm"
Nov 03 09:54:30.923356 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.923334    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5"
Nov 03 09:54:30.926827 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.926809    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh"
Nov 03 09:54:30.946677 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.946649    4473 generic.go:356] "Generic (PLEG): container finished" podID="60b1d7ef-16c6-4b6a-973b-4a251593eac7" containerID="3744118d3b2020f3e6089947695f142df33af61398f13df40b08c08432f9f451" exitCode=0
Nov 03 09:54:30.946796 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.946680    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm"
Nov 03 09:54:30.946796 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.946705    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm" event={"ID":"60b1d7ef-16c6-4b6a-973b-4a251593eac7","Type":"ContainerDied","Data":"3744118d3b2020f3e6089947695f142df33af61398f13df40b08c08432f9f451"}
Nov 03 09:54:30.946796 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.946742    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm" event={"ID":"60b1d7ef-16c6-4b6a-973b-4a251593eac7","Type":"ContainerDied","Data":"a8e5579320c4ee4c2aa165782f3dae107b74cac51a871041a4fa8590c65d12d3"}
Nov 03 09:54:30.946796 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.946759    4473 scope.go:122] "RemoveContainer" containerID="3744118d3b2020f3e6089947695f142df33af61398f13df40b08c08432f9f451"
Nov 03 09:54:30.948259 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.948236    4473 generic.go:356] "Generic (PLEG): container finished" podID="2e42501e-d39d-4ae1-91c7-76c3c171538e" containerID="804a116da52cdfcc4d7e8f31ff5aae521e8c0d7291a469edb1b72c1fa18a1a4c" exitCode=0
Nov 03 09:54:30.948370 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.948282    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5"
Nov 03 09:54:30.948370 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.948332    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5" event={"ID":"2e42501e-d39d-4ae1-91c7-76c3c171538e","Type":"ContainerDied","Data":"804a116da52cdfcc4d7e8f31ff5aae521e8c0d7291a469edb1b72c1fa18a1a4c"}
Nov 03 09:54:30.948463 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.948373    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5" event={"ID":"2e42501e-d39d-4ae1-91c7-76c3c171538e","Type":"ContainerDied","Data":"66115fc418a0df9ccb64e9712d438e688b99d36801b9636615504d757f287565"}
Nov 03 09:54:30.949805 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.949787    4473 generic.go:356] "Generic (PLEG): container finished" podID="9e7f39b3-ce66-4b2d-abbc-8f078d527a2d" containerID="6cbfa3600b6ec31caa148e0864bef52b0805a33bb415ff10a049c4154e11b8f8" exitCode=0
Nov 03 09:54:30.949906 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.949819    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g"
Nov 03 09:54:30.949906 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.949856    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g" event={"ID":"9e7f39b3-ce66-4b2d-abbc-8f078d527a2d","Type":"ContainerDied","Data":"6cbfa3600b6ec31caa148e0864bef52b0805a33bb415ff10a049c4154e11b8f8"}
Nov 03 09:54:30.949906 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.949893    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g" event={"ID":"9e7f39b3-ce66-4b2d-abbc-8f078d527a2d","Type":"ContainerDied","Data":"e18c2d60643d0f54d6675596cc7507d7f178de0c3b1654df7247a88b430276f2"}
Nov 03 09:54:30.951218 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.951193    4473 generic.go:356] "Generic (PLEG): container finished" podID="00bab831-c667-415d-b6b0-81fff4d57762" containerID="3a357ddbfa22890aeeb1111586d18167a593964ad6d736077e6169c86223dbcf" exitCode=0
Nov 03 09:54:30.951315 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.951229    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh"
Nov 03 09:54:30.951315 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.951239    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh" event={"ID":"00bab831-c667-415d-b6b0-81fff4d57762","Type":"ContainerDied","Data":"3a357ddbfa22890aeeb1111586d18167a593964ad6d736077e6169c86223dbcf"}
Nov 03 09:54:30.951315 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.951274    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh" event={"ID":"00bab831-c667-415d-b6b0-81fff4d57762","Type":"ContainerDied","Data":"604db5aa29f63682b2415a9fc345178b03517c82574178fe585aa1a4b02d537d"}
Nov 03 09:54:30.956534 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.956513    4473 scope.go:122] "RemoveContainer" containerID="3744118d3b2020f3e6089947695f142df33af61398f13df40b08c08432f9f451"
Nov 03 09:54:30.956785 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:54:30.956768    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3744118d3b2020f3e6089947695f142df33af61398f13df40b08c08432f9f451\": container with ID starting with 3744118d3b2020f3e6089947695f142df33af61398f13df40b08c08432f9f451 not found: ID does not exist" containerID="3744118d3b2020f3e6089947695f142df33af61398f13df40b08c08432f9f451"
Nov 03 09:54:30.956841 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.956792    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3744118d3b2020f3e6089947695f142df33af61398f13df40b08c08432f9f451"} err="failed to get container status \"3744118d3b2020f3e6089947695f142df33af61398f13df40b08c08432f9f451\": rpc error: code = NotFound desc = could not find container \"3744118d3b2020f3e6089947695f142df33af61398f13df40b08c08432f9f451\": container with ID starting with 3744118d3b2020f3e6089947695f142df33af61398f13df40b08c08432f9f451 not found: ID does not exist"
Nov 03 09:54:30.956841 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.956807    4473 scope.go:122] "RemoveContainer" containerID="804a116da52cdfcc4d7e8f31ff5aae521e8c0d7291a469edb1b72c1fa18a1a4c"
Nov 03 09:54:30.965669 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.965650    4473 scope.go:122] "RemoveContainer" containerID="804a116da52cdfcc4d7e8f31ff5aae521e8c0d7291a469edb1b72c1fa18a1a4c"
Nov 03 09:54:30.965925 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:54:30.965907    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"804a116da52cdfcc4d7e8f31ff5aae521e8c0d7291a469edb1b72c1fa18a1a4c\": container with ID starting with 804a116da52cdfcc4d7e8f31ff5aae521e8c0d7291a469edb1b72c1fa18a1a4c not found: ID does not exist" containerID="804a116da52cdfcc4d7e8f31ff5aae521e8c0d7291a469edb1b72c1fa18a1a4c"
Nov 03 09:54:30.965996 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.965933    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"804a116da52cdfcc4d7e8f31ff5aae521e8c0d7291a469edb1b72c1fa18a1a4c"} err="failed to get container status \"804a116da52cdfcc4d7e8f31ff5aae521e8c0d7291a469edb1b72c1fa18a1a4c\": rpc error: code = NotFound desc = could not find container \"804a116da52cdfcc4d7e8f31ff5aae521e8c0d7291a469edb1b72c1fa18a1a4c\": container with ID starting with 804a116da52cdfcc4d7e8f31ff5aae521e8c0d7291a469edb1b72c1fa18a1a4c not found: ID does not exist"
Nov 03 09:54:30.965996 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.965967    4473 scope.go:122] "RemoveContainer" containerID="6cbfa3600b6ec31caa148e0864bef52b0805a33bb415ff10a049c4154e11b8f8"
Nov 03 09:54:30.974447 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.974423    4473 scope.go:122] "RemoveContainer" containerID="6cbfa3600b6ec31caa148e0864bef52b0805a33bb415ff10a049c4154e11b8f8"
Nov 03 09:54:30.974721 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:54:30.974701    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cbfa3600b6ec31caa148e0864bef52b0805a33bb415ff10a049c4154e11b8f8\": container with ID starting with 6cbfa3600b6ec31caa148e0864bef52b0805a33bb415ff10a049c4154e11b8f8 not found: ID does not exist" containerID="6cbfa3600b6ec31caa148e0864bef52b0805a33bb415ff10a049c4154e11b8f8"
Nov 03 09:54:30.974786 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.974727    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cbfa3600b6ec31caa148e0864bef52b0805a33bb415ff10a049c4154e11b8f8"} err="failed to get container status \"6cbfa3600b6ec31caa148e0864bef52b0805a33bb415ff10a049c4154e11b8f8\": rpc error: code = NotFound desc = could not find container \"6cbfa3600b6ec31caa148e0864bef52b0805a33bb415ff10a049c4154e11b8f8\": container with ID starting with 6cbfa3600b6ec31caa148e0864bef52b0805a33bb415ff10a049c4154e11b8f8 not found: ID does not exist"
Nov 03 09:54:30.974786 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.974745    4473 scope.go:122] "RemoveContainer" containerID="3a357ddbfa22890aeeb1111586d18167a593964ad6d736077e6169c86223dbcf"
Nov 03 09:54:30.985522 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.985502    4473 scope.go:122] "RemoveContainer" containerID="3a357ddbfa22890aeeb1111586d18167a593964ad6d736077e6169c86223dbcf"
Nov 03 09:54:30.986041 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:54:30.985931    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a357ddbfa22890aeeb1111586d18167a593964ad6d736077e6169c86223dbcf\": container with ID starting with 3a357ddbfa22890aeeb1111586d18167a593964ad6d736077e6169c86223dbcf not found: ID does not exist" containerID="3a357ddbfa22890aeeb1111586d18167a593964ad6d736077e6169c86223dbcf"
Nov 03 09:54:30.986041 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.985997    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a357ddbfa22890aeeb1111586d18167a593964ad6d736077e6169c86223dbcf"} err="failed to get container status \"3a357ddbfa22890aeeb1111586d18167a593964ad6d736077e6169c86223dbcf\": rpc error: code = NotFound desc = could not find container \"3a357ddbfa22890aeeb1111586d18167a593964ad6d736077e6169c86223dbcf\": container with ID starting with 3a357ddbfa22890aeeb1111586d18167a593964ad6d736077e6169c86223dbcf not found: ID does not exist"
Nov 03 09:54:30.999696 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.999672    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60b1d7ef-16c6-4b6a-973b-4a251593eac7-kube-api-access-gtwkq\" (UniqueName: \"kubernetes.io/projected/60b1d7ef-16c6-4b6a-973b-4a251593eac7-kube-api-access-gtwkq\") pod \"60b1d7ef-16c6-4b6a-973b-4a251593eac7\" (UID: \"60b1d7ef-16c6-4b6a-973b-4a251593eac7\") "
Nov 03 09:54:30.999811 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.999724    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00bab831-c667-415d-b6b0-81fff4d57762-kube-api-access-bqjlv\" (UniqueName: \"kubernetes.io/projected/00bab831-c667-415d-b6b0-81fff4d57762-kube-api-access-bqjlv\") pod \"00bab831-c667-415d-b6b0-81fff4d57762\" (UID: \"00bab831-c667-415d-b6b0-81fff4d57762\") "
Nov 03 09:54:30.999811 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.999750    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00bab831-c667-415d-b6b0-81fff4d57762-home\" (UniqueName: \"kubernetes.io/empty-dir/00bab831-c667-415d-b6b0-81fff4d57762-home\") pod \"00bab831-c667-415d-b6b0-81fff4d57762\" (UID: \"00bab831-c667-415d-b6b0-81fff4d57762\") "
Nov 03 09:54:30.999811 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.999781    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00bab831-c667-415d-b6b0-81fff4d57762-tls-certs\" (UniqueName: \"kubernetes.io/secret/00bab831-c667-415d-b6b0-81fff4d57762-tls-certs\") pod \"00bab831-c667-415d-b6b0-81fff4d57762\" (UID: \"00bab831-c667-415d-b6b0-81fff4d57762\") "
Nov 03 09:54:30.999811 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.999805    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00bab831-c667-415d-b6b0-81fff4d57762-dshm\" (UniqueName: \"kubernetes.io/empty-dir/00bab831-c667-415d-b6b0-81fff4d57762-dshm\") pod \"00bab831-c667-415d-b6b0-81fff4d57762\" (UID: \"00bab831-c667-415d-b6b0-81fff4d57762\") "
Nov 03 09:54:31.000148 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.999835    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e42501e-d39d-4ae1-91c7-76c3c171538e-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e42501e-d39d-4ae1-91c7-76c3c171538e-tls-certs\") pod \"2e42501e-d39d-4ae1-91c7-76c3c171538e\" (UID: \"2e42501e-d39d-4ae1-91c7-76c3c171538e\") "
Nov 03 09:54:31.000148 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.999868    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e42501e-d39d-4ae1-91c7-76c3c171538e-home\" (UniqueName: \"kubernetes.io/empty-dir/2e42501e-d39d-4ae1-91c7-76c3c171538e-home\") pod \"2e42501e-d39d-4ae1-91c7-76c3c171538e\" (UID: \"2e42501e-d39d-4ae1-91c7-76c3c171538e\") "
Nov 03 09:54:31.000148 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.999914    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-tls-certs\") pod \"9e7f39b3-ce66-4b2d-abbc-8f078d527a2d\" (UID: \"9e7f39b3-ce66-4b2d-abbc-8f078d527a2d\") "
Nov 03 09:54:31.000148 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.999932    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e42501e-d39d-4ae1-91c7-76c3c171538e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e42501e-d39d-4ae1-91c7-76c3c171538e-model-cache\") pod \"2e42501e-d39d-4ae1-91c7-76c3c171538e\" (UID: \"2e42501e-d39d-4ae1-91c7-76c3c171538e\") "
Nov 03 09:54:31.000148 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:30.999989    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60b1d7ef-16c6-4b6a-973b-4a251593eac7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/60b1d7ef-16c6-4b6a-973b-4a251593eac7-dshm\") pod \"60b1d7ef-16c6-4b6a-973b-4a251593eac7\" (UID: \"60b1d7ef-16c6-4b6a-973b-4a251593eac7\") "
Nov 03 09:54:31.000431 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.000204    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e42501e-d39d-4ae1-91c7-76c3c171538e-home" pod "2e42501e-d39d-4ae1-91c7-76c3c171538e" (UID: "2e42501e-d39d-4ae1-91c7-76c3c171538e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:54:31.002087 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.000734    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00bab831-c667-415d-b6b0-81fff4d57762-home" pod "00bab831-c667-415d-b6b0-81fff4d57762" (UID: "00bab831-c667-415d-b6b0-81fff4d57762"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:54:31.002087 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.000962    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e42501e-d39d-4ae1-91c7-76c3c171538e-model-cache" pod "2e42501e-d39d-4ae1-91c7-76c3c171538e" (UID: "2e42501e-d39d-4ae1-91c7-76c3c171538e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:54:31.002087 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.001010    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e42501e-d39d-4ae1-91c7-76c3c171538e-kube-api-access-zw4mh\" (UniqueName: \"kubernetes.io/projected/2e42501e-d39d-4ae1-91c7-76c3c171538e-kube-api-access-zw4mh\") pod \"2e42501e-d39d-4ae1-91c7-76c3c171538e\" (UID: \"2e42501e-d39d-4ae1-91c7-76c3c171538e\") "
Nov 03 09:54:31.002087 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.001062    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-kube-api-access-rlvth\" (UniqueName: \"kubernetes.io/projected/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-kube-api-access-rlvth\") pod \"9e7f39b3-ce66-4b2d-abbc-8f078d527a2d\" (UID: \"9e7f39b3-ce66-4b2d-abbc-8f078d527a2d\") "
Nov 03 09:54:31.002087 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.001099    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60b1d7ef-16c6-4b6a-973b-4a251593eac7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60b1d7ef-16c6-4b6a-973b-4a251593eac7-tmp-dir\") pod \"60b1d7ef-16c6-4b6a-973b-4a251593eac7\" (UID: \"60b1d7ef-16c6-4b6a-973b-4a251593eac7\") "
Nov 03 09:54:31.002087 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.001136    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60b1d7ef-16c6-4b6a-973b-4a251593eac7-home\" (UniqueName: \"kubernetes.io/empty-dir/60b1d7ef-16c6-4b6a-973b-4a251593eac7-home\") pod \"60b1d7ef-16c6-4b6a-973b-4a251593eac7\" (UID: \"60b1d7ef-16c6-4b6a-973b-4a251593eac7\") "
Nov 03 09:54:31.002087 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.001167    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e42501e-d39d-4ae1-91c7-76c3c171538e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e42501e-d39d-4ae1-91c7-76c3c171538e-tmp-dir\") pod \"2e42501e-d39d-4ae1-91c7-76c3c171538e\" (UID: \"2e42501e-d39d-4ae1-91c7-76c3c171538e\") "
Nov 03 09:54:31.002087 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.001204    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00bab831-c667-415d-b6b0-81fff4d57762-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00bab831-c667-415d-b6b0-81fff4d57762-tmp-dir\") pod \"00bab831-c667-415d-b6b0-81fff4d57762\" (UID: \"00bab831-c667-415d-b6b0-81fff4d57762\") "
Nov 03 09:54:31.002087 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.001269    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-tmp-dir\") pod \"9e7f39b3-ce66-4b2d-abbc-8f078d527a2d\" (UID: \"9e7f39b3-ce66-4b2d-abbc-8f078d527a2d\") "
Nov 03 09:54:31.002087 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.001314    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-dshm\") pod \"9e7f39b3-ce66-4b2d-abbc-8f078d527a2d\" (UID: \"9e7f39b3-ce66-4b2d-abbc-8f078d527a2d\") "
Nov 03 09:54:31.002087 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.001414    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e42501e-d39d-4ae1-91c7-76c3c171538e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e42501e-d39d-4ae1-91c7-76c3c171538e-dshm\") pod \"2e42501e-d39d-4ae1-91c7-76c3c171538e\" (UID: \"2e42501e-d39d-4ae1-91c7-76c3c171538e\") "
Nov 03 09:54:31.002087 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.001450    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-home\" (UniqueName: \"kubernetes.io/empty-dir/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-home\") pod \"9e7f39b3-ce66-4b2d-abbc-8f078d527a2d\" (UID: \"9e7f39b3-ce66-4b2d-abbc-8f078d527a2d\") "
Nov 03 09:54:31.002087 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.001500    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-model-cache\") pod \"9e7f39b3-ce66-4b2d-abbc-8f078d527a2d\" (UID: \"9e7f39b3-ce66-4b2d-abbc-8f078d527a2d\") "
Nov 03 09:54:31.002087 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.001551    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00bab831-c667-415d-b6b0-81fff4d57762-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00bab831-c667-415d-b6b0-81fff4d57762-model-cache\") pod \"00bab831-c667-415d-b6b0-81fff4d57762\" (UID: \"00bab831-c667-415d-b6b0-81fff4d57762\") "
Nov 03 09:54:31.002087 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.001613    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60b1d7ef-16c6-4b6a-973b-4a251593eac7-tls-certs\" (UniqueName: \"kubernetes.io/secret/60b1d7ef-16c6-4b6a-973b-4a251593eac7-tls-certs\") pod \"60b1d7ef-16c6-4b6a-973b-4a251593eac7\" (UID: \"60b1d7ef-16c6-4b6a-973b-4a251593eac7\") "
Nov 03 09:54:31.002087 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.001653    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60b1d7ef-16c6-4b6a-973b-4a251593eac7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60b1d7ef-16c6-4b6a-973b-4a251593eac7-model-cache\") pod \"60b1d7ef-16c6-4b6a-973b-4a251593eac7\" (UID: \"60b1d7ef-16c6-4b6a-973b-4a251593eac7\") "
Nov 03 09:54:31.002087 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.001857    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b1d7ef-16c6-4b6a-973b-4a251593eac7-tmp-dir" pod "60b1d7ef-16c6-4b6a-973b-4a251593eac7" (UID: "60b1d7ef-16c6-4b6a-973b-4a251593eac7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:54:31.003468 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.002312    4473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00bab831-c667-415d-b6b0-81fff4d57762-home\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:31.003468 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.002333    4473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e42501e-d39d-4ae1-91c7-76c3c171538e-home\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:31.003468 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.002348    4473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e42501e-d39d-4ae1-91c7-76c3c171538e-model-cache\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:31.003468 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.002362    4473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60b1d7ef-16c6-4b6a-973b-4a251593eac7-tmp-dir\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:31.004431 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.003731    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b1d7ef-16c6-4b6a-973b-4a251593eac7-model-cache" pod "60b1d7ef-16c6-4b6a-973b-4a251593eac7" (UID: "60b1d7ef-16c6-4b6a-973b-4a251593eac7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:54:31.004431 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.003972    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-model-cache" pod "9e7f39b3-ce66-4b2d-abbc-8f078d527a2d" (UID: "9e7f39b3-ce66-4b2d-abbc-8f078d527a2d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:54:31.004431 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.004035    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b1d7ef-16c6-4b6a-973b-4a251593eac7-home" pod "60b1d7ef-16c6-4b6a-973b-4a251593eac7" (UID: "60b1d7ef-16c6-4b6a-973b-4a251593eac7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:54:31.004431 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.004161    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00bab831-c667-415d-b6b0-81fff4d57762-model-cache" pod "00bab831-c667-415d-b6b0-81fff4d57762" (UID: "00bab831-c667-415d-b6b0-81fff4d57762"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:54:31.004431 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.004299    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e42501e-d39d-4ae1-91c7-76c3c171538e-tmp-dir" pod "2e42501e-d39d-4ae1-91c7-76c3c171538e" (UID: "2e42501e-d39d-4ae1-91c7-76c3c171538e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:54:31.004810 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.004505    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00bab831-c667-415d-b6b0-81fff4d57762-tmp-dir" pod "00bab831-c667-415d-b6b0-81fff4d57762" (UID: "00bab831-c667-415d-b6b0-81fff4d57762"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:54:31.004810 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.004698    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-tmp-dir" pod "9e7f39b3-ce66-4b2d-abbc-8f078d527a2d" (UID: "9e7f39b3-ce66-4b2d-abbc-8f078d527a2d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:54:31.004810 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.004760    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-home" pod "9e7f39b3-ce66-4b2d-abbc-8f078d527a2d" (UID: "9e7f39b3-ce66-4b2d-abbc-8f078d527a2d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:54:31.005010 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.004816    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e42501e-d39d-4ae1-91c7-76c3c171538e-tls-certs" pod "2e42501e-d39d-4ae1-91c7-76c3c171538e" (UID: "2e42501e-d39d-4ae1-91c7-76c3c171538e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:54:31.006145 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.006115    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60b1d7ef-16c6-4b6a-973b-4a251593eac7-kube-api-access-gtwkq" pod "60b1d7ef-16c6-4b6a-973b-4a251593eac7" (UID: "60b1d7ef-16c6-4b6a-973b-4a251593eac7"). InnerVolumeSpecName "kube-api-access-gtwkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:54:31.006626 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.006605    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e42501e-d39d-4ae1-91c7-76c3c171538e-dshm" pod "2e42501e-d39d-4ae1-91c7-76c3c171538e" (UID: "2e42501e-d39d-4ae1-91c7-76c3c171538e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:54:31.006783 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.006661    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-dshm" pod "9e7f39b3-ce66-4b2d-abbc-8f078d527a2d" (UID: "9e7f39b3-ce66-4b2d-abbc-8f078d527a2d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:54:31.007547 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.007508    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e42501e-d39d-4ae1-91c7-76c3c171538e-kube-api-access-zw4mh" pod "2e42501e-d39d-4ae1-91c7-76c3c171538e" (UID: "2e42501e-d39d-4ae1-91c7-76c3c171538e"). InnerVolumeSpecName "kube-api-access-zw4mh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:54:31.007803 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.007775    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60b1d7ef-16c6-4b6a-973b-4a251593eac7-dshm" pod "60b1d7ef-16c6-4b6a-973b-4a251593eac7" (UID: "60b1d7ef-16c6-4b6a-973b-4a251593eac7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:54:31.009311 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.009287    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00bab831-c667-415d-b6b0-81fff4d57762-kube-api-access-bqjlv" pod "00bab831-c667-415d-b6b0-81fff4d57762" (UID: "00bab831-c667-415d-b6b0-81fff4d57762"). InnerVolumeSpecName "kube-api-access-bqjlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:54:31.009461 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.009442    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00bab831-c667-415d-b6b0-81fff4d57762-dshm" pod "00bab831-c667-415d-b6b0-81fff4d57762" (UID: "00bab831-c667-415d-b6b0-81fff4d57762"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:54:31.009550 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.009504    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-tls-certs" pod "9e7f39b3-ce66-4b2d-abbc-8f078d527a2d" (UID: "9e7f39b3-ce66-4b2d-abbc-8f078d527a2d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:54:31.009781 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.009760    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00bab831-c667-415d-b6b0-81fff4d57762-tls-certs" pod "00bab831-c667-415d-b6b0-81fff4d57762" (UID: "00bab831-c667-415d-b6b0-81fff4d57762"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:54:31.009923 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.009905    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-kube-api-access-rlvth" pod "9e7f39b3-ce66-4b2d-abbc-8f078d527a2d" (UID: "9e7f39b3-ce66-4b2d-abbc-8f078d527a2d"). InnerVolumeSpecName "kube-api-access-rlvth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:54:31.011230 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.011211    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60b1d7ef-16c6-4b6a-973b-4a251593eac7-tls-certs" pod "60b1d7ef-16c6-4b6a-973b-4a251593eac7" (UID: "60b1d7ef-16c6-4b6a-973b-4a251593eac7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:54:31.103678 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.103601    4473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60b1d7ef-16c6-4b6a-973b-4a251593eac7-model-cache\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:31.103678 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.103639    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtwkq\" (UniqueName: \"kubernetes.io/projected/60b1d7ef-16c6-4b6a-973b-4a251593eac7-kube-api-access-gtwkq\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:31.103678 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.103655    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqjlv\" (UniqueName: \"kubernetes.io/projected/00bab831-c667-415d-b6b0-81fff4d57762-kube-api-access-bqjlv\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:31.103678 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.103670    4473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00bab831-c667-415d-b6b0-81fff4d57762-tls-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:31.103924 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.103685    4473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00bab831-c667-415d-b6b0-81fff4d57762-dshm\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:31.103924 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.103698    4473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e42501e-d39d-4ae1-91c7-76c3c171538e-tls-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:31.103924 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.103711    4473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-tls-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:31.103924 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.103723    4473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60b1d7ef-16c6-4b6a-973b-4a251593eac7-dshm\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:31.103924 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.103736    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw4mh\" (UniqueName: \"kubernetes.io/projected/2e42501e-d39d-4ae1-91c7-76c3c171538e-kube-api-access-zw4mh\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:31.103924 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.103750    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlvth\" (UniqueName: \"kubernetes.io/projected/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-kube-api-access-rlvth\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:31.103924 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.103765    4473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60b1d7ef-16c6-4b6a-973b-4a251593eac7-home\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:31.103924 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.103777    4473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e42501e-d39d-4ae1-91c7-76c3c171538e-tmp-dir\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:31.103924 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.103791    4473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00bab831-c667-415d-b6b0-81fff4d57762-tmp-dir\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:31.103924 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.103801    4473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-tmp-dir\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:31.103924 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.103813    4473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-dshm\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:31.103924 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.103826    4473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e42501e-d39d-4ae1-91c7-76c3c171538e-dshm\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:31.103924 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.103839    4473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-home\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:31.103924 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.103853    4473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d-model-cache\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:31.103924 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.103866    4473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00bab831-c667-415d-b6b0-81fff4d57762-model-cache\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:31.103924 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.103879    4473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60b1d7ef-16c6-4b6a-973b-4a251593eac7-tls-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:31.494539 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.494508    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g"]
Nov 03 09:54:31.499790 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.499764    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g"]
Nov 03 09:54:31.611046 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.611015    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm"]
Nov 03 09:54:31.618867 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.618840    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-18f9da/premium-w4-18f9da-kserve-5d68844675-5jpkm"]
Nov 03 09:54:31.729655 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.729623    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh"]
Nov 03 09:54:31.735025 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.735003    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-18f9da/simulator-w4-18f9da-kserve-66dd7c494b-x4zlh"]
Nov 03 09:54:31.845772 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.845708    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-18f9da/distinct-2-w4-18f9da-kserve-76b55d979c-5cv5g" podUID="9e7f39b3-ce66-4b2d-abbc-8f078d527a2d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 09:54:31.846348 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.846331    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5"]
Nov 03 09:54:31.852002 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.851976    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-18f9da/distinct-w4-18f9da-kserve-65fccfbbf8-6kfk5"]
Nov 03 09:54:31.869424 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.869390    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00bab831-c667-415d-b6b0-81fff4d57762" path="/var/lib/kubelet/pods/00bab831-c667-415d-b6b0-81fff4d57762/volumes"
Nov 03 09:54:31.869885 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.869865    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e42501e-d39d-4ae1-91c7-76c3c171538e" path="/var/lib/kubelet/pods/2e42501e-d39d-4ae1-91c7-76c3c171538e/volumes"
Nov 03 09:54:31.870348 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.870329    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60b1d7ef-16c6-4b6a-973b-4a251593eac7" path="/var/lib/kubelet/pods/60b1d7ef-16c6-4b6a-973b-4a251593eac7/volumes"
Nov 03 09:54:31.871069 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:31.871044    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e7f39b3-ce66-4b2d-abbc-8f078d527a2d" path="/var/lib/kubelet/pods/9e7f39b3-ce66-4b2d-abbc-8f078d527a2d/volumes"
Nov 03 09:54:32.368297 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:32.368265    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56e07e4a-lc726"]
Nov 03 09:54:32.368711 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:32.368698    4473 cpu_manager.go:414] "containerMap: removing container" podUID="60b1d7ef-16c6-4b6a-973b-4a251593eac7" containerName="main"
Nov 03 09:54:32.368755 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:32.368712    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60b1d7ef-16c6-4b6a-973b-4a251593eac7" containerName="main"
Nov 03 09:54:32.368755 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:32.368725    4473 cpu_manager.go:414] "containerMap: removing container" podUID="00bab831-c667-415d-b6b0-81fff4d57762" containerName="main"
Nov 03 09:54:32.368755 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:32.368729    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00bab831-c667-415d-b6b0-81fff4d57762" containerName="main"
Nov 03 09:54:32.368755 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:32.368742    4473 cpu_manager.go:414] "containerMap: removing container" podUID="2e42501e-d39d-4ae1-91c7-76c3c171538e" containerName="main"
Nov 03 09:54:32.368755 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:32.368747    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e42501e-d39d-4ae1-91c7-76c3c171538e" containerName="main"
Nov 03 09:54:32.368755 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:32.368754    4473 cpu_manager.go:414] "containerMap: removing container" podUID="73981a22-201c-42d3-b1d6-81afe3c9d24f" containerName="revoke-keys"
Nov 03 09:54:32.368980 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:32.368759    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73981a22-201c-42d3-b1d6-81afe3c9d24f" containerName="revoke-keys"
Nov 03 09:54:32.368980 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:32.368785    4473 cpu_manager.go:414] "containerMap: removing container" podUID="89f277d9-a48e-4fbe-b866-b26ef10468d4" containerName="revoke-keys"
Nov 03 09:54:32.368980 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:32.368790    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89f277d9-a48e-4fbe-b866-b26ef10468d4" containerName="revoke-keys"
Nov 03 09:54:32.368980 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:32.368798    4473 cpu_manager.go:414] "containerMap: removing container" podUID="9e7f39b3-ce66-4b2d-abbc-8f078d527a2d" containerName="main"
Nov 03 09:54:32.368980 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:32.368802    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e7f39b3-ce66-4b2d-abbc-8f078d527a2d" containerName="main"
Nov 03 09:54:32.368980 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:32.368867    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e42501e-d39d-4ae1-91c7-76c3c171538e" containerName="main"
Nov 03 09:54:32.368980 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:32.368874    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="73981a22-201c-42d3-b1d6-81afe3c9d24f" containerName="revoke-keys"
Nov 03 09:54:32.368980 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:32.368885    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="89f277d9-a48e-4fbe-b866-b26ef10468d4" containerName="revoke-keys"
Nov 03 09:54:32.368980 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:32.368891    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e7f39b3-ce66-4b2d-abbc-8f078d527a2d" containerName="main"
Nov 03 09:54:32.368980 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:32.368898    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="60b1d7ef-16c6-4b6a-973b-4a251593eac7" containerName="main"
Nov 03 09:54:32.368980 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:32.368903    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="00bab831-c667-415d-b6b0-81fff4d57762" containerName="main"
Nov 03 09:54:32.379198 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:32.379171    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e07e4a-lc726"
Nov 03 09:54:32.429598 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:32.429570    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56e07e4a-lc726"]
Nov 03 09:54:32.434443 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:32.434417    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:54:32.434586 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:32.434505    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:54:32.434648 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:32.434598    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:54:32.434648 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:32.434635    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:54:32.518444 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:32.518411    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45e6234a-28a0-47ff-a266-815869ef9666-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56e07e4a-lc726\" (UID: \"45e6234a-28a0-47ff-a266-815869ef9666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e07e4a-lc726"
Nov 03 09:54:32.518629 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:32.518465    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45e6234a-28a0-47ff-a266-815869ef9666-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56e07e4a-lc726\" (UID: \"45e6234a-28a0-47ff-a266-815869ef9666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e07e4a-lc726"
Nov 03 09:54:32.518629 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:32.518610    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbr9k\" (UniqueName: \"kubernetes.io/projected/45e6234a-28a0-47ff-a266-815869ef9666-kube-api-access-kbr9k\") pod \"maas-api-revoke-sub-56e07e4a-lc726\" (UID: \"45e6234a-28a0-47ff-a266-815869ef9666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e07e4a-lc726"
Nov 03 09:54:32.619693 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:32.619618    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45e6234a-28a0-47ff-a266-815869ef9666-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56e07e4a-lc726\" (UID: \"45e6234a-28a0-47ff-a266-815869ef9666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e07e4a-lc726"
Nov 03 09:54:32.619693 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:32.619683    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbr9k\" (UniqueName: \"kubernetes.io/projected/45e6234a-28a0-47ff-a266-815869ef9666-kube-api-access-kbr9k\") pod \"maas-api-revoke-sub-56e07e4a-lc726\" (UID: \"45e6234a-28a0-47ff-a266-815869ef9666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e07e4a-lc726"
Nov 03 09:54:32.620190 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:32.619763    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45e6234a-28a0-47ff-a266-815869ef9666-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56e07e4a-lc726\" (UID: \"45e6234a-28a0-47ff-a266-815869ef9666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e07e4a-lc726"
Nov 03 09:54:32.620323 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:32.620299    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45e6234a-28a0-47ff-a266-815869ef9666-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56e07e4a-lc726\" (UID: \"45e6234a-28a0-47ff-a266-815869ef9666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e07e4a-lc726"
Nov 03 09:54:32.622189 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:32.622165    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45e6234a-28a0-47ff-a266-815869ef9666-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56e07e4a-lc726\" (UID: \"45e6234a-28a0-47ff-a266-815869ef9666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e07e4a-lc726"
Nov 03 09:54:32.678845 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:32.678820    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbr9k\" (UniqueName: \"kubernetes.io/projected/45e6234a-28a0-47ff-a266-815869ef9666-kube-api-access-kbr9k\") pod \"maas-api-revoke-sub-56e07e4a-lc726\" (UID: \"45e6234a-28a0-47ff-a266-815869ef9666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e07e4a-lc726"
Nov 03 09:54:32.691532 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:32.691513    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e07e4a-lc726"
Nov 03 09:54:33.066764 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:33.066738    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56e07e4a-lc726"]
Nov 03 09:54:33.067574 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:54:33.067545    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e6234a_28a0_47ff_a266_815869ef9666.slice/crio-e16b88df9eff72fd0cd214530148ccb78fa082c98aae8630ea53ce13d0f3d533 WatchSource:0}: Error finding container e16b88df9eff72fd0cd214530148ccb78fa082c98aae8630ea53ce13d0f3d533: Status 404 returned error can't find the container with id e16b88df9eff72fd0cd214530148ccb78fa082c98aae8630ea53ce13d0f3d533
Nov 03 09:54:33.973691 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:33.973653    4473 generic.go:356] "Generic (PLEG): container finished" podID="45e6234a-28a0-47ff-a266-815869ef9666" containerID="f2f1da200b6e3038534a4f417a87f0512f72cd5f0243d9f841511fd97590de0f" exitCode=0
Nov 03 09:54:33.974127 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:33.973739    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e07e4a-lc726" event={"ID":"45e6234a-28a0-47ff-a266-815869ef9666","Type":"ContainerDied","Data":"f2f1da200b6e3038534a4f417a87f0512f72cd5f0243d9f841511fd97590de0f"}
Nov 03 09:54:33.974127 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:33.973780    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e07e4a-lc726" event={"ID":"45e6234a-28a0-47ff-a266-815869ef9666","Type":"ContainerStarted","Data":"e16b88df9eff72fd0cd214530148ccb78fa082c98aae8630ea53ce13d0f3d533"}
Nov 03 09:54:34.352228 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:34.352145    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"]
Nov 03 09:54:34.355614 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:34.355599    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"
Nov 03 09:54:34.438663 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:34.438632    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-istio-token\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5\" (UID: \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"
Nov 03 09:54:34.438812 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:34.438667    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzh4c\" (UniqueName: \"kubernetes.io/projected/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-kube-api-access-xzh4c\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5\" (UID: \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"
Nov 03 09:54:34.438812 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:34.438689    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-istiod-ca-cert\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5\" (UID: \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"
Nov 03 09:54:34.439002 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:34.438814    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-credential-socket\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5\" (UID: \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"
Nov 03 09:54:34.439002 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:34.438846    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-workload-socket\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5\" (UID: \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"
Nov 03 09:54:34.439002 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:34.438872    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-istio-podinfo\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5\" (UID: \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"
Nov 03 09:54:34.439002 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:34.438978    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-istio-envoy\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5\" (UID: \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"
Nov 03 09:54:34.439002 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:34.438999    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-istio-data\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5\" (UID: \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"
Nov 03 09:54:34.439176 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:34.439026    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-workload-certs\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5\" (UID: \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"
Nov 03 09:54:34.461590 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:34.461560    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-b196c8-openshift-default-dockercfg-hrsmg\""
Nov 03 09:54:34.464443 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:34.464424    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"]
Nov 03 09:54:34.539682 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:34.539650    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-istio-envoy\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5\" (UID: \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"
Nov 03 09:54:34.539812 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:34.539689    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-istio-data\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5\" (UID: \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"
Nov 03 09:54:34.539869 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:34.539821    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-workload-certs\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5\" (UID: \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"
Nov 03 09:54:34.539914 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:34.539881    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-istio-token\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5\" (UID: \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"
Nov 03 09:54:34.540005 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:34.539921    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzh4c\" (UniqueName: \"kubernetes.io/projected/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-kube-api-access-xzh4c\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5\" (UID: \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"
Nov 03 09:54:34.540005 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:34.539999    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-istiod-ca-cert\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5\" (UID: \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"
Nov 03 09:54:34.540120 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:34.540081    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-istio-data\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5\" (UID: \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"
Nov 03 09:54:34.540120 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:34.540092    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-credential-socket\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5\" (UID: \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"
Nov 03 09:54:34.540221 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:34.540153    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-workload-socket\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5\" (UID: \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"
Nov 03 09:54:34.540221 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:34.540191    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-workload-certs\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5\" (UID: \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"
Nov 03 09:54:34.540221 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:34.540199    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-istio-podinfo\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5\" (UID: \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"
Nov 03 09:54:34.540392 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:34.540372    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-credential-socket\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5\" (UID: \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"
Nov 03 09:54:34.540459 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:34.540443    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-workload-socket\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5\" (UID: \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"
Nov 03 09:54:34.540742 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:34.540720    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-istiod-ca-cert\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5\" (UID: \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"
Nov 03 09:54:34.542273 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:34.542254    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-istio-envoy\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5\" (UID: \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"
Nov 03 09:54:34.542624 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:34.542608    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-istio-podinfo\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5\" (UID: \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"
Nov 03 09:54:34.600428 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:34.600404    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-istio-token\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5\" (UID: \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"
Nov 03 09:54:34.600558 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:34.600541    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzh4c\" (UniqueName: \"kubernetes.io/projected/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-kube-api-access-xzh4c\") pod \"e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5\" (UID: \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\") " pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"
Nov 03 09:54:34.668601 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:34.668581    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"
Nov 03 09:54:35.097739 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:35.097706    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"]
Nov 03 09:54:35.128209 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:35.128186    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e07e4a-lc726"
Nov 03 09:54:35.135678 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:54:35.135656    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda697a58e_ba2e_4bff_a9fb_0d749cc6e044.slice/crio-f2d0ae0697693af44d7af7babd5730cd9235ba078cd42f407839a81e94b37ad3 WatchSource:0}: Error finding container f2d0ae0697693af44d7af7babd5730cd9235ba078cd42f407839a81e94b37ad3: Status 404 returned error can't find the container with id f2d0ae0697693af44d7af7babd5730cd9235ba078cd42f407839a81e94b37ad3
Nov 03 09:54:35.138034 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:35.138004    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:54:35.138097 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:35.138070    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:54:35.138137 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:35.138098    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 09:54:35.249086 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:35.246643    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45e6234a-28a0-47ff-a266-815869ef9666-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45e6234a-28a0-47ff-a266-815869ef9666-maas-api-openshift-ca\") pod \"45e6234a-28a0-47ff-a266-815869ef9666\" (UID: \"45e6234a-28a0-47ff-a266-815869ef9666\") "
Nov 03 09:54:35.249086 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:35.246770    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45e6234a-28a0-47ff-a266-815869ef9666-kube-api-access-kbr9k\" (UniqueName: \"kubernetes.io/projected/45e6234a-28a0-47ff-a266-815869ef9666-kube-api-access-kbr9k\") pod \"45e6234a-28a0-47ff-a266-815869ef9666\" (UID: \"45e6234a-28a0-47ff-a266-815869ef9666\") "
Nov 03 09:54:35.249086 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:35.246868    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45e6234a-28a0-47ff-a266-815869ef9666-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45e6234a-28a0-47ff-a266-815869ef9666-maas-api-xks-ca\") pod \"45e6234a-28a0-47ff-a266-815869ef9666\" (UID: \"45e6234a-28a0-47ff-a266-815869ef9666\") "
Nov 03 09:54:35.249086 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:35.247677    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45e6234a-28a0-47ff-a266-815869ef9666-maas-api-openshift-ca" pod "45e6234a-28a0-47ff-a266-815869ef9666" (UID: "45e6234a-28a0-47ff-a266-815869ef9666"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:54:35.250867 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:35.250828    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45e6234a-28a0-47ff-a266-815869ef9666-kube-api-access-kbr9k" pod "45e6234a-28a0-47ff-a266-815869ef9666" (UID: "45e6234a-28a0-47ff-a266-815869ef9666"). InnerVolumeSpecName "kube-api-access-kbr9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:54:35.251739 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:35.251695    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45e6234a-28a0-47ff-a266-815869ef9666-maas-api-xks-ca" pod "45e6234a-28a0-47ff-a266-815869ef9666" (UID: "45e6234a-28a0-47ff-a266-815869ef9666"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:54:35.348606 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:35.348568    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbr9k\" (UniqueName: \"kubernetes.io/projected/45e6234a-28a0-47ff-a266-815869ef9666-kube-api-access-kbr9k\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:35.348606 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:35.348605    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45e6234a-28a0-47ff-a266-815869ef9666-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:35.348820 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:35.348621    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45e6234a-28a0-47ff-a266-815869ef9666-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:35.944384 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:35.944294    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:54:35.986478 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:35.986454    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e07e4a-lc726"
Nov 03 09:54:35.986635 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:35.986502    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56e07e4a-lc726" event={"ID":"45e6234a-28a0-47ff-a266-815869ef9666","Type":"ContainerDied","Data":"e16b88df9eff72fd0cd214530148ccb78fa082c98aae8630ea53ce13d0f3d533"}
Nov 03 09:54:35.986635 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:35.986547    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e16b88df9eff72fd0cd214530148ccb78fa082c98aae8630ea53ce13d0f3d533"
Nov 03 09:54:35.988449 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:35.988423    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5" event={"ID":"a697a58e-ba2e-4bff-a9fb-0d749cc6e044","Type":"ContainerStarted","Data":"921e998ef623e19db1396991c181984dfe4df354c68b7384184f5eff2c857ebc"}
Nov 03 09:54:35.988644 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:35.988471    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5" event={"ID":"a697a58e-ba2e-4bff-a9fb-0d749cc6e044","Type":"ContainerStarted","Data":"f2d0ae0697693af44d7af7babd5730cd9235ba078cd42f407839a81e94b37ad3"}
Nov 03 09:54:36.110841 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:36.110792    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5" podStartSLOduration=2.110776898 podStartE2EDuration="2.110776898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:54:36.109445976 +0000 UTC m=+4500.378509412" watchObservedRunningTime="2026-11-03 09:54:36.110776898 +0000 UTC m=+4500.379840331"
Nov 03 09:54:36.669319 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:36.669287    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"
Nov 03 09:54:36.670710 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:36.670689    4473 patch_prober.go:29] interesting pod/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Nov 03 09:54:36.670788 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:36.670736    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5" podUID="a697a58e-ba2e-4bff-a9fb-0d749cc6e044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Nov 03 09:54:36.871444 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:36.871415    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 09:54:36.871620 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:36.871540    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 09:54:37.669160 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:37.669124    4473 patch_prober.go:29] interesting pod/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Nov 03 09:54:37.669526 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:37.669197    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5" podUID="a697a58e-ba2e-4bff-a9fb-0d749cc6e044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Nov 03 09:54:38.669843 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:38.669804    4473 patch_prober.go:29] interesting pod/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Nov 03 09:54:38.670255 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:38.669864    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5" podUID="a697a58e-ba2e-4bff-a9fb-0d749cc6e044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Nov 03 09:54:39.669473 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:39.669432    4473 patch_prober.go:29] interesting pod/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Nov 03 09:54:39.669652 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:39.669493    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5" podUID="a697a58e-ba2e-4bff-a9fb-0d749cc6e044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Nov 03 09:54:40.669600 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:40.669561    4473 patch_prober.go:29] interesting pod/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Nov 03 09:54:40.670042 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:40.669626    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5" podUID="a697a58e-ba2e-4bff-a9fb-0d749cc6e044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Nov 03 09:54:41.739100 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:41.739051    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"
Nov 03 09:54:41.739857 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:41.739827    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"
Nov 03 09:54:41.740878 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:41.740858    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"
Nov 03 09:54:44.006963 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:44.006907    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc1db4ce-6hvxg"]
Nov 03 09:54:44.007405 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:44.007391    4473 cpu_manager.go:414] "containerMap: removing container" podUID="45e6234a-28a0-47ff-a266-815869ef9666" containerName="revoke-keys"
Nov 03 09:54:44.007453 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:44.007405    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45e6234a-28a0-47ff-a266-815869ef9666" containerName="revoke-keys"
Nov 03 09:54:44.007503 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:44.007494    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="45e6234a-28a0-47ff-a266-815869ef9666" containerName="revoke-keys"
Nov 03 09:54:44.017860 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:44.017839    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1db4ce-6hvxg"
Nov 03 09:54:44.046566 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:44.046535    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krbqc\" (UniqueName: \"kubernetes.io/projected/55d63835-33f6-4b7d-8d7d-894ebc43a641-kube-api-access-krbqc\") pod \"maas-api-revoke-sub-dc1db4ce-6hvxg\" (UID: \"55d63835-33f6-4b7d-8d7d-894ebc43a641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1db4ce-6hvxg"
Nov 03 09:54:44.046738 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:44.046712    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55d63835-33f6-4b7d-8d7d-894ebc43a641-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc1db4ce-6hvxg\" (UID: \"55d63835-33f6-4b7d-8d7d-894ebc43a641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1db4ce-6hvxg"
Nov 03 09:54:44.046856 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:44.046827    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55d63835-33f6-4b7d-8d7d-894ebc43a641-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc1db4ce-6hvxg\" (UID: \"55d63835-33f6-4b7d-8d7d-894ebc43a641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1db4ce-6hvxg"
Nov 03 09:54:44.069130 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:44.069104    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc1db4ce-6hvxg"]
Nov 03 09:54:44.115712 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:44.115689    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:54:44.115840 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:44.115747    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:54:44.121642 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:44.121621    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:54:44.121781 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:44.121767    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:54:44.147540 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:44.147511    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55d63835-33f6-4b7d-8d7d-894ebc43a641-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc1db4ce-6hvxg\" (UID: \"55d63835-33f6-4b7d-8d7d-894ebc43a641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1db4ce-6hvxg"
Nov 03 09:54:44.147648 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:44.147584    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55d63835-33f6-4b7d-8d7d-894ebc43a641-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc1db4ce-6hvxg\" (UID: \"55d63835-33f6-4b7d-8d7d-894ebc43a641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1db4ce-6hvxg"
Nov 03 09:54:44.147648 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:44.147624    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krbqc\" (UniqueName: \"kubernetes.io/projected/55d63835-33f6-4b7d-8d7d-894ebc43a641-kube-api-access-krbqc\") pod \"maas-api-revoke-sub-dc1db4ce-6hvxg\" (UID: \"55d63835-33f6-4b7d-8d7d-894ebc43a641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1db4ce-6hvxg"
Nov 03 09:54:44.148294 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:44.148272    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55d63835-33f6-4b7d-8d7d-894ebc43a641-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc1db4ce-6hvxg\" (UID: \"55d63835-33f6-4b7d-8d7d-894ebc43a641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1db4ce-6hvxg"
Nov 03 09:54:44.150034 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:44.150018    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55d63835-33f6-4b7d-8d7d-894ebc43a641-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc1db4ce-6hvxg\" (UID: \"55d63835-33f6-4b7d-8d7d-894ebc43a641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1db4ce-6hvxg"
Nov 03 09:54:44.206763 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:44.206738    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krbqc\" (UniqueName: \"kubernetes.io/projected/55d63835-33f6-4b7d-8d7d-894ebc43a641-kube-api-access-krbqc\") pod \"maas-api-revoke-sub-dc1db4ce-6hvxg\" (UID: \"55d63835-33f6-4b7d-8d7d-894ebc43a641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1db4ce-6hvxg"
Nov 03 09:54:44.331431 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:44.331358    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1db4ce-6hvxg"
Nov 03 09:54:44.705749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:44.705720    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc1db4ce-6hvxg"]
Nov 03 09:54:44.706614 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:54:44.706588    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55d63835_33f6_4b7d_8d7d_894ebc43a641.slice/crio-a27104f3c92ccb3378fc8a72219f9fb30741e1f76fa22dcdd3deeabedbaa9546 WatchSource:0}: Error finding container a27104f3c92ccb3378fc8a72219f9fb30741e1f76fa22dcdd3deeabedbaa9546: Status 404 returned error can't find the container with id a27104f3c92ccb3378fc8a72219f9fb30741e1f76fa22dcdd3deeabedbaa9546
Nov 03 09:54:45.043138 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:45.043110    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1db4ce-6hvxg" event={"ID":"55d63835-33f6-4b7d-8d7d-894ebc43a641","Type":"ContainerStarted","Data":"523a0a5a38007e0eb7723f85a12de1e2cae431309429c213382f0c27ada5dac6"}
Nov 03 09:54:45.043502 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:45.043160    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1db4ce-6hvxg" event={"ID":"55d63835-33f6-4b7d-8d7d-894ebc43a641","Type":"ContainerStarted","Data":"a27104f3c92ccb3378fc8a72219f9fb30741e1f76fa22dcdd3deeabedbaa9546"}
Nov 03 09:54:45.160632 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:45.160574    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1db4ce-6hvxg" podStartSLOduration=2.16055976 podStartE2EDuration="2.16055976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:54:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:54:45.158923634 +0000 UTC m=+4509.427987090" watchObservedRunningTime="2026-11-03 09:54:45.16055976 +0000 UTC m=+4509.429623222"
Nov 03 09:54:46.052873 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:46.052834    4473 generic.go:356] "Generic (PLEG): container finished" podID="55d63835-33f6-4b7d-8d7d-894ebc43a641" containerID="523a0a5a38007e0eb7723f85a12de1e2cae431309429c213382f0c27ada5dac6" exitCode=0
Nov 03 09:54:46.053305 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:46.052928    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1db4ce-6hvxg" event={"ID":"55d63835-33f6-4b7d-8d7d-894ebc43a641","Type":"ContainerDied","Data":"523a0a5a38007e0eb7723f85a12de1e2cae431309429c213382f0c27ada5dac6"}
Nov 03 09:54:46.943140 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:54:46.943103    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:54:47.245637 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:47.245608    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1db4ce-6hvxg"
Nov 03 09:54:47.277611 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:47.277573    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55d63835-33f6-4b7d-8d7d-894ebc43a641-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55d63835-33f6-4b7d-8d7d-894ebc43a641-maas-api-xks-ca\") pod \"55d63835-33f6-4b7d-8d7d-894ebc43a641\" (UID: \"55d63835-33f6-4b7d-8d7d-894ebc43a641\") "
Nov 03 09:54:47.277802 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:47.277688    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55d63835-33f6-4b7d-8d7d-894ebc43a641-kube-api-access-krbqc\" (UniqueName: \"kubernetes.io/projected/55d63835-33f6-4b7d-8d7d-894ebc43a641-kube-api-access-krbqc\") pod \"55d63835-33f6-4b7d-8d7d-894ebc43a641\" (UID: \"55d63835-33f6-4b7d-8d7d-894ebc43a641\") "
Nov 03 09:54:47.277802 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:47.277729    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55d63835-33f6-4b7d-8d7d-894ebc43a641-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55d63835-33f6-4b7d-8d7d-894ebc43a641-maas-api-openshift-ca\") pod \"55d63835-33f6-4b7d-8d7d-894ebc43a641\" (UID: \"55d63835-33f6-4b7d-8d7d-894ebc43a641\") "
Nov 03 09:54:47.278261 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:47.278231    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55d63835-33f6-4b7d-8d7d-894ebc43a641-maas-api-openshift-ca" pod "55d63835-33f6-4b7d-8d7d-894ebc43a641" (UID: "55d63835-33f6-4b7d-8d7d-894ebc43a641"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:54:47.279971 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:47.279928    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55d63835-33f6-4b7d-8d7d-894ebc43a641-maas-api-xks-ca" pod "55d63835-33f6-4b7d-8d7d-894ebc43a641" (UID: "55d63835-33f6-4b7d-8d7d-894ebc43a641"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:54:47.280059 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:47.280036    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55d63835-33f6-4b7d-8d7d-894ebc43a641-kube-api-access-krbqc" pod "55d63835-33f6-4b7d-8d7d-894ebc43a641" (UID: "55d63835-33f6-4b7d-8d7d-894ebc43a641"). InnerVolumeSpecName "kube-api-access-krbqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:54:47.378813 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:47.378783    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55d63835-33f6-4b7d-8d7d-894ebc43a641-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:47.378813 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:47.378809    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55d63835-33f6-4b7d-8d7d-894ebc43a641-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:47.378813 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:47.378822    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krbqc\" (UniqueName: \"kubernetes.io/projected/55d63835-33f6-4b7d-8d7d-894ebc43a641-kube-api-access-krbqc\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:48.086560 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:48.086524    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1db4ce-6hvxg"
Nov 03 09:54:48.086743 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:48.086554    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1db4ce-6hvxg" event={"ID":"55d63835-33f6-4b7d-8d7d-894ebc43a641","Type":"ContainerDied","Data":"a27104f3c92ccb3378fc8a72219f9fb30741e1f76fa22dcdd3deeabedbaa9546"}
Nov 03 09:54:48.086743 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:48.086613    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a27104f3c92ccb3378fc8a72219f9fb30741e1f76fa22dcdd3deeabedbaa9546"
Nov 03 09:54:54.979697 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:54.979669    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57788ad7-p2m5c"]
Nov 03 09:54:54.980175 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:54.980161    4473 cpu_manager.go:414] "containerMap: removing container" podUID="55d63835-33f6-4b7d-8d7d-894ebc43a641" containerName="revoke-keys"
Nov 03 09:54:54.980224 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:54.980176    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55d63835-33f6-4b7d-8d7d-894ebc43a641" containerName="revoke-keys"
Nov 03 09:54:54.980279 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:54.980264    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="55d63835-33f6-4b7d-8d7d-894ebc43a641" containerName="revoke-keys"
Nov 03 09:54:54.983689 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:54.983673    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57788ad7-p2m5c"
Nov 03 09:54:55.041158 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:55.041129    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57788ad7-p2m5c"]
Nov 03 09:54:55.073241 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:55.073201    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5605651d-4be1-4b2c-a4a3-827f6fd13146-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57788ad7-p2m5c\" (UID: \"5605651d-4be1-4b2c-a4a3-827f6fd13146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57788ad7-p2m5c"
Nov 03 09:54:55.073425 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:55.073256    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5605651d-4be1-4b2c-a4a3-827f6fd13146-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57788ad7-p2m5c\" (UID: \"5605651d-4be1-4b2c-a4a3-827f6fd13146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57788ad7-p2m5c"
Nov 03 09:54:55.073425 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:55.073399    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg2vk\" (UniqueName: \"kubernetes.io/projected/5605651d-4be1-4b2c-a4a3-827f6fd13146-kube-api-access-qg2vk\") pod \"maas-api-revoke-sub-57788ad7-p2m5c\" (UID: \"5605651d-4be1-4b2c-a4a3-827f6fd13146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57788ad7-p2m5c"
Nov 03 09:54:55.087689 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:55.087665    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:54:55.087689 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:55.087665    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:54:55.138079 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:55.138045    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:54:55.138251 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:55.138145    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:54:55.174676 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:55.174641    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5605651d-4be1-4b2c-a4a3-827f6fd13146-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57788ad7-p2m5c\" (UID: \"5605651d-4be1-4b2c-a4a3-827f6fd13146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57788ad7-p2m5c"
Nov 03 09:54:55.174853 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:55.174733    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg2vk\" (UniqueName: \"kubernetes.io/projected/5605651d-4be1-4b2c-a4a3-827f6fd13146-kube-api-access-qg2vk\") pod \"maas-api-revoke-sub-57788ad7-p2m5c\" (UID: \"5605651d-4be1-4b2c-a4a3-827f6fd13146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57788ad7-p2m5c"
Nov 03 09:54:55.174853 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:55.174804    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5605651d-4be1-4b2c-a4a3-827f6fd13146-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57788ad7-p2m5c\" (UID: \"5605651d-4be1-4b2c-a4a3-827f6fd13146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57788ad7-p2m5c"
Nov 03 09:54:55.175486 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:55.175463    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5605651d-4be1-4b2c-a4a3-827f6fd13146-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57788ad7-p2m5c\" (UID: \"5605651d-4be1-4b2c-a4a3-827f6fd13146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57788ad7-p2m5c"
Nov 03 09:54:55.177194 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:55.177170    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5605651d-4be1-4b2c-a4a3-827f6fd13146-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57788ad7-p2m5c\" (UID: \"5605651d-4be1-4b2c-a4a3-827f6fd13146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57788ad7-p2m5c"
Nov 03 09:54:55.233872 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:55.233799    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg2vk\" (UniqueName: \"kubernetes.io/projected/5605651d-4be1-4b2c-a4a3-827f6fd13146-kube-api-access-qg2vk\") pod \"maas-api-revoke-sub-57788ad7-p2m5c\" (UID: \"5605651d-4be1-4b2c-a4a3-827f6fd13146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57788ad7-p2m5c"
Nov 03 09:54:55.296620 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:55.296586    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57788ad7-p2m5c"
Nov 03 09:54:55.672268 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:55.672237    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57788ad7-p2m5c"]
Nov 03 09:54:55.673681 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:54:55.673654    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5605651d_4be1_4b2c_a4a3_827f6fd13146.slice/crio-eb5fe3d253a8df8fc2dd4b088f596af55ede9593e02b9e130f403117f1b62dc9 WatchSource:0}: Error finding container eb5fe3d253a8df8fc2dd4b088f596af55ede9593e02b9e130f403117f1b62dc9: Status 404 returned error can't find the container with id eb5fe3d253a8df8fc2dd4b088f596af55ede9593e02b9e130f403117f1b62dc9
Nov 03 09:54:56.139745 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:56.139706    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57788ad7-p2m5c" event={"ID":"5605651d-4be1-4b2c-a4a3-827f6fd13146","Type":"ContainerStarted","Data":"fb206098c0fead159851f1854855bdedc485622108ee717fbf416eea18e0e27d"}
Nov 03 09:54:56.140136 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:56.139768    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57788ad7-p2m5c" event={"ID":"5605651d-4be1-4b2c-a4a3-827f6fd13146","Type":"ContainerStarted","Data":"eb5fe3d253a8df8fc2dd4b088f596af55ede9593e02b9e130f403117f1b62dc9"}
Nov 03 09:54:56.256817 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:56.256713    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57788ad7-p2m5c" podStartSLOduration=2.256698995 podStartE2EDuration="2.256698995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:54:56.255028426 +0000 UTC m=+4520.524091883" watchObservedRunningTime="2026-11-03 09:54:56.256698995 +0000 UTC m=+4520.525762407"
Nov 03 09:54:57.147204 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:57.147165    4473 generic.go:356] "Generic (PLEG): container finished" podID="5605651d-4be1-4b2c-a4a3-827f6fd13146" containerID="fb206098c0fead159851f1854855bdedc485622108ee717fbf416eea18e0e27d" exitCode=0
Nov 03 09:54:57.147595 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:57.147248    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57788ad7-p2m5c" event={"ID":"5605651d-4be1-4b2c-a4a3-827f6fd13146","Type":"ContainerDied","Data":"fb206098c0fead159851f1854855bdedc485622108ee717fbf416eea18e0e27d"}
Nov 03 09:54:58.312214 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:58.312185    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57788ad7-p2m5c"
Nov 03 09:54:58.406584 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:58.406475    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5605651d-4be1-4b2c-a4a3-827f6fd13146-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5605651d-4be1-4b2c-a4a3-827f6fd13146-maas-api-xks-ca\") pod \"5605651d-4be1-4b2c-a4a3-827f6fd13146\" (UID: \"5605651d-4be1-4b2c-a4a3-827f6fd13146\") "
Nov 03 09:54:58.406584 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:58.406570    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5605651d-4be1-4b2c-a4a3-827f6fd13146-kube-api-access-qg2vk\" (UniqueName: \"kubernetes.io/projected/5605651d-4be1-4b2c-a4a3-827f6fd13146-kube-api-access-qg2vk\") pod \"5605651d-4be1-4b2c-a4a3-827f6fd13146\" (UID: \"5605651d-4be1-4b2c-a4a3-827f6fd13146\") "
Nov 03 09:54:58.406849 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:58.406649    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5605651d-4be1-4b2c-a4a3-827f6fd13146-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5605651d-4be1-4b2c-a4a3-827f6fd13146-maas-api-openshift-ca\") pod \"5605651d-4be1-4b2c-a4a3-827f6fd13146\" (UID: \"5605651d-4be1-4b2c-a4a3-827f6fd13146\") "
Nov 03 09:54:58.409247 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:58.407975    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5605651d-4be1-4b2c-a4a3-827f6fd13146-maas-api-openshift-ca" pod "5605651d-4be1-4b2c-a4a3-827f6fd13146" (UID: "5605651d-4be1-4b2c-a4a3-827f6fd13146"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:54:58.419709 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:58.419568    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5605651d-4be1-4b2c-a4a3-827f6fd13146-kube-api-access-qg2vk" pod "5605651d-4be1-4b2c-a4a3-827f6fd13146" (UID: "5605651d-4be1-4b2c-a4a3-827f6fd13146"). InnerVolumeSpecName "kube-api-access-qg2vk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:54:58.420013 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:58.419876    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5605651d-4be1-4b2c-a4a3-827f6fd13146-maas-api-xks-ca" pod "5605651d-4be1-4b2c-a4a3-827f6fd13146" (UID: "5605651d-4be1-4b2c-a4a3-827f6fd13146"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:54:58.511228 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:58.511182    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg2vk\" (UniqueName: \"kubernetes.io/projected/5605651d-4be1-4b2c-a4a3-827f6fd13146-kube-api-access-qg2vk\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:58.511509 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:58.511485    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5605651d-4be1-4b2c-a4a3-827f6fd13146-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:58.511655 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:58.511638    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5605651d-4be1-4b2c-a4a3-827f6fd13146-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:54:59.163380 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:59.163342    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57788ad7-p2m5c" event={"ID":"5605651d-4be1-4b2c-a4a3-827f6fd13146","Type":"ContainerDied","Data":"eb5fe3d253a8df8fc2dd4b088f596af55ede9593e02b9e130f403117f1b62dc9"}
Nov 03 09:54:59.163566 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:59.163406    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb5fe3d253a8df8fc2dd4b088f596af55ede9593e02b9e130f403117f1b62dc9"
Nov 03 09:54:59.163566 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:54:59.163442    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57788ad7-p2m5c"
Nov 03 09:55:06.993795 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:06.993763    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-f9b517-6f6ff64d89-qd7zj"]
Nov 03 09:55:07.100563 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:07.100530    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f9b517-646c9dd495-qt8s8"]
Nov 03 09:55:11.994736 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:11.994657    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f9b517-6f6ff64d89-qd7zj" podUID="c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9" containerName="payload-processing" containerID="cri-o://6b4eec6fcb2b235eb0db416abe970e09d2ef562e5ff2471f948db643aeb7a425" gracePeriod=25
Nov 03 09:55:12.101748 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:12.101696    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f9b517-646c9dd495-qt8s8" podUID="842a3cc8-4062-41df-aa2f-18c19bb342c9" containerName="payload-pre-processing" containerID="cri-o://f2c9e77ac698d682bd952dc6b5c6ae98b7472e003a30d56b08871a6e62fbb0b9" gracePeriod=25
Nov 03 09:55:16.737371 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:16.737327    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f9b517-646c9dd495-qt8s8" podUID="842a3cc8-4062-41df-aa2f-18c19bb342c9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:55:16.737807 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:16.737579    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f9b517-6f6ff64d89-qd7zj" podUID="c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:55:16.937026 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:55:16.936926    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:55:23.554968 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:23.554914    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:55:26.737474 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:26.737430    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f9b517-646c9dd495-qt8s8" podUID="842a3cc8-4062-41df-aa2f-18c19bb342c9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:55:26.739762 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:26.737528    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f9b517-6f6ff64d89-qd7zj" podUID="c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:55:32.265755 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:32.265687    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f9b517-6f6ff64d89-qd7zj"
Nov 03 09:55:32.356513 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:32.356489    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f9b517-646c9dd495-qt8s8"
Nov 03 09:55:32.368489 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:32.368464    4473 generic.go:356] "Generic (PLEG): container finished" podID="842a3cc8-4062-41df-aa2f-18c19bb342c9" containerID="f2c9e77ac698d682bd952dc6b5c6ae98b7472e003a30d56b08871a6e62fbb0b9" exitCode=0
Nov 03 09:55:32.368635 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:32.368502    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f9b517-646c9dd495-qt8s8"
Nov 03 09:55:32.368635 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:32.368548    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f9b517-646c9dd495-qt8s8" event={"ID":"842a3cc8-4062-41df-aa2f-18c19bb342c9","Type":"ContainerDied","Data":"f2c9e77ac698d682bd952dc6b5c6ae98b7472e003a30d56b08871a6e62fbb0b9"}
Nov 03 09:55:32.368635 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:32.368602    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f9b517-646c9dd495-qt8s8" event={"ID":"842a3cc8-4062-41df-aa2f-18c19bb342c9","Type":"ContainerDied","Data":"1b8666259751eb49f0e47c93be9b61dbcdfd6a4f3f3357c76526341b67ba4eec"}
Nov 03 09:55:32.368635 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:32.368630    4473 scope.go:122] "RemoveContainer" containerID="f2c9e77ac698d682bd952dc6b5c6ae98b7472e003a30d56b08871a6e62fbb0b9"
Nov 03 09:55:32.369987 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:32.369926    4473 generic.go:356] "Generic (PLEG): container finished" podID="c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9" containerID="6b4eec6fcb2b235eb0db416abe970e09d2ef562e5ff2471f948db643aeb7a425" exitCode=0
Nov 03 09:55:32.369987 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:32.369974    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f9b517-6f6ff64d89-qd7zj" event={"ID":"c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9","Type":"ContainerDied","Data":"6b4eec6fcb2b235eb0db416abe970e09d2ef562e5ff2471f948db643aeb7a425"}
Nov 03 09:55:32.369987 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:32.370019    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f9b517-6f6ff64d89-qd7zj" event={"ID":"c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9","Type":"ContainerDied","Data":"a703cfba713907cab1d77a55eb5d0d8b9a4cf0117354737c772de95c3f91ae95"}
Nov 03 09:55:32.369987 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:32.369979    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f9b517-6f6ff64d89-qd7zj"
Nov 03 09:55:32.377763 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:32.377749    4473 scope.go:122] "RemoveContainer" containerID="f2c9e77ac698d682bd952dc6b5c6ae98b7472e003a30d56b08871a6e62fbb0b9"
Nov 03 09:55:32.378034 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:55:32.378016    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2c9e77ac698d682bd952dc6b5c6ae98b7472e003a30d56b08871a6e62fbb0b9\": container with ID starting with f2c9e77ac698d682bd952dc6b5c6ae98b7472e003a30d56b08871a6e62fbb0b9 not found: ID does not exist" containerID="f2c9e77ac698d682bd952dc6b5c6ae98b7472e003a30d56b08871a6e62fbb0b9"
Nov 03 09:55:32.378085 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:32.378041    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2c9e77ac698d682bd952dc6b5c6ae98b7472e003a30d56b08871a6e62fbb0b9"} err="failed to get container status \"f2c9e77ac698d682bd952dc6b5c6ae98b7472e003a30d56b08871a6e62fbb0b9\": rpc error: code = NotFound desc = could not find container \"f2c9e77ac698d682bd952dc6b5c6ae98b7472e003a30d56b08871a6e62fbb0b9\": container with ID starting with f2c9e77ac698d682bd952dc6b5c6ae98b7472e003a30d56b08871a6e62fbb0b9 not found: ID does not exist"
Nov 03 09:55:32.378085 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:32.378056    4473 scope.go:122] "RemoveContainer" containerID="6b4eec6fcb2b235eb0db416abe970e09d2ef562e5ff2471f948db643aeb7a425"
Nov 03 09:55:32.388576 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:32.388555    4473 scope.go:122] "RemoveContainer" containerID="6b4eec6fcb2b235eb0db416abe970e09d2ef562e5ff2471f948db643aeb7a425"
Nov 03 09:55:32.388906 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:55:32.388868    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b4eec6fcb2b235eb0db416abe970e09d2ef562e5ff2471f948db643aeb7a425\": container with ID starting with 6b4eec6fcb2b235eb0db416abe970e09d2ef562e5ff2471f948db643aeb7a425 not found: ID does not exist" containerID="6b4eec6fcb2b235eb0db416abe970e09d2ef562e5ff2471f948db643aeb7a425"
Nov 03 09:55:32.389043 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:32.388913    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b4eec6fcb2b235eb0db416abe970e09d2ef562e5ff2471f948db643aeb7a425"} err="failed to get container status \"6b4eec6fcb2b235eb0db416abe970e09d2ef562e5ff2471f948db643aeb7a425\": rpc error: code = NotFound desc = could not find container \"6b4eec6fcb2b235eb0db416abe970e09d2ef562e5ff2471f948db643aeb7a425\": container with ID starting with 6b4eec6fcb2b235eb0db416abe970e09d2ef562e5ff2471f948db643aeb7a425 not found: ID does not exist"
Nov 03 09:55:32.410769 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:32.410745    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9-plugins-config-volume\") pod \"c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9\" (UID: \"c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9\") "
Nov 03 09:55:32.410913 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:32.410828    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9-kube-api-access-lpjnw\" (UniqueName: \"kubernetes.io/projected/c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9-kube-api-access-lpjnw\") pod \"c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9\" (UID: \"c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9\") "
Nov 03 09:55:32.411143 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:32.411122    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9-plugins-config-volume" pod "c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9" (UID: "c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:55:32.413100 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:32.413076    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9-kube-api-access-lpjnw" pod "c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9" (UID: "c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9"). InnerVolumeSpecName "kube-api-access-lpjnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:55:32.512414 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:32.512391    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/842a3cc8-4062-41df-aa2f-18c19bb342c9-kube-api-access-cpzls\" (UniqueName: \"kubernetes.io/projected/842a3cc8-4062-41df-aa2f-18c19bb342c9-kube-api-access-cpzls\") pod \"842a3cc8-4062-41df-aa2f-18c19bb342c9\" (UID: \"842a3cc8-4062-41df-aa2f-18c19bb342c9\") "
Nov 03 09:55:32.512560 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:32.512482    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/842a3cc8-4062-41df-aa2f-18c19bb342c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/842a3cc8-4062-41df-aa2f-18c19bb342c9-plugins-config-volume\") pod \"842a3cc8-4062-41df-aa2f-18c19bb342c9\" (UID: \"842a3cc8-4062-41df-aa2f-18c19bb342c9\") "
Nov 03 09:55:32.512820 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:32.512799    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/842a3cc8-4062-41df-aa2f-18c19bb342c9-plugins-config-volume" pod "842a3cc8-4062-41df-aa2f-18c19bb342c9" (UID: "842a3cc8-4062-41df-aa2f-18c19bb342c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:55:32.512889 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:32.512861    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:55:32.512889 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:32.512876    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpjnw\" (UniqueName: \"kubernetes.io/projected/c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9-kube-api-access-lpjnw\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:55:32.514656 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:32.514635    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/842a3cc8-4062-41df-aa2f-18c19bb342c9-kube-api-access-cpzls" pod "842a3cc8-4062-41df-aa2f-18c19bb342c9" (UID: "842a3cc8-4062-41df-aa2f-18c19bb342c9"). InnerVolumeSpecName "kube-api-access-cpzls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:55:32.613606 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:32.613532    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpzls\" (UniqueName: \"kubernetes.io/projected/842a3cc8-4062-41df-aa2f-18c19bb342c9-kube-api-access-cpzls\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:55:32.613606 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:32.613560    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/842a3cc8-4062-41df-aa2f-18c19bb342c9-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:55:32.797402 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:32.797370    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f9b517-646c9dd495-qt8s8"]
Nov 03 09:55:32.803836 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:32.803811    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f9b517-646c9dd495-qt8s8"]
Nov 03 09:55:32.915819 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:32.915785    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-f9b517-6f6ff64d89-qd7zj"]
Nov 03 09:55:32.920808 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:32.920784    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-f9b517-6f6ff64d89-qd7zj"]
Nov 03 09:55:33.870328 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:33.870298    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="842a3cc8-4062-41df-aa2f-18c19bb342c9" path="/var/lib/kubelet/pods/842a3cc8-4062-41df-aa2f-18c19bb342c9/volumes"
Nov 03 09:55:33.870843 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:33.870806    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9" path="/var/lib/kubelet/pods/c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9/volumes"
Nov 03 09:55:37.652752 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:37.652717    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-627e5f/sub-test-model-627e5f-kserve-6785c9fd7f-z4chr"]
Nov 03 09:55:40.548510 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:40.548475    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b196c8/sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77"]
Nov 03 09:55:46.938019 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:55:46.937979    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:55:50.054106 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:50.054069    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18a7390b-h2tcj"]
Nov 03 09:55:50.104021 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:50.103992    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18a7390b-h2tcj"]
Nov 03 09:55:51.870226 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:51.870192    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92f3682c-4935-4a79-824a-6db5f2ac5c53" path="/var/lib/kubelet/pods/92f3682c-4935-4a79-824a-6db5f2ac5c53/volumes"
Nov 03 09:55:52.677339 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:52.677300    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-627e5f/sub-test-model-627e5f-kserve-6785c9fd7f-z4chr" podUID="9bf62832-a048-47e7-91d5-cb4eb34e7e26" containerName="main" containerID="cri-o://72890e92d941f1a7147c0a2b7f5fc300bc442c8200fe08b9b4fe9b5bb464ae24" gracePeriod=45
Nov 03 09:55:52.852393 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:52.852350    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-627e5f/sub-test-model-627e5f-kserve-6785c9fd7f-z4chr" podUID="9bf62832-a048-47e7-91d5-cb4eb34e7e26" containerName="main" probeResult="failure" output="Get \"https://10.133.0.227:8000/ready\": dial tcp 10.133.0.227:8000: connect: connection refused"
Nov 03 09:55:52.941409 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:52.941330    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-627e5f/sub-test-model-627e5f-kserve-6785c9fd7f-z4chr"
Nov 03 09:55:53.072828 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:53.072797    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bf62832-a048-47e7-91d5-cb4eb34e7e26-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bf62832-a048-47e7-91d5-cb4eb34e7e26-dshm\") pod \"9bf62832-a048-47e7-91d5-cb4eb34e7e26\" (UID: \"9bf62832-a048-47e7-91d5-cb4eb34e7e26\") "
Nov 03 09:55:53.073047 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:53.072859    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bf62832-a048-47e7-91d5-cb4eb34e7e26-home\" (UniqueName: \"kubernetes.io/empty-dir/9bf62832-a048-47e7-91d5-cb4eb34e7e26-home\") pod \"9bf62832-a048-47e7-91d5-cb4eb34e7e26\" (UID: \"9bf62832-a048-47e7-91d5-cb4eb34e7e26\") "
Nov 03 09:55:53.073047 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:53.072932    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bf62832-a048-47e7-91d5-cb4eb34e7e26-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bf62832-a048-47e7-91d5-cb4eb34e7e26-model-cache\") pod \"9bf62832-a048-47e7-91d5-cb4eb34e7e26\" (UID: \"9bf62832-a048-47e7-91d5-cb4eb34e7e26\") "
Nov 03 09:55:53.073047 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:53.073009    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bf62832-a048-47e7-91d5-cb4eb34e7e26-tls-certs\" (UniqueName: \"kubernetes.io/secret/9bf62832-a048-47e7-91d5-cb4eb34e7e26-tls-certs\") pod \"9bf62832-a048-47e7-91d5-cb4eb34e7e26\" (UID: \"9bf62832-a048-47e7-91d5-cb4eb34e7e26\") "
Nov 03 09:55:53.073266 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:53.073059    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bf62832-a048-47e7-91d5-cb4eb34e7e26-kube-api-access-szqxc\" (UniqueName: \"kubernetes.io/projected/9bf62832-a048-47e7-91d5-cb4eb34e7e26-kube-api-access-szqxc\") pod \"9bf62832-a048-47e7-91d5-cb4eb34e7e26\" (UID: \"9bf62832-a048-47e7-91d5-cb4eb34e7e26\") "
Nov 03 09:55:53.073266 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:53.073109    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bf62832-a048-47e7-91d5-cb4eb34e7e26-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bf62832-a048-47e7-91d5-cb4eb34e7e26-tmp-dir\") pod \"9bf62832-a048-47e7-91d5-cb4eb34e7e26\" (UID: \"9bf62832-a048-47e7-91d5-cb4eb34e7e26\") "
Nov 03 09:55:53.073389 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:53.073323    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bf62832-a048-47e7-91d5-cb4eb34e7e26-model-cache" pod "9bf62832-a048-47e7-91d5-cb4eb34e7e26" (UID: "9bf62832-a048-47e7-91d5-cb4eb34e7e26"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:55:53.073560 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:53.073537    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bf62832-a048-47e7-91d5-cb4eb34e7e26-tmp-dir" pod "9bf62832-a048-47e7-91d5-cb4eb34e7e26" (UID: "9bf62832-a048-47e7-91d5-cb4eb34e7e26"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:55:53.073749 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:53.073702    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bf62832-a048-47e7-91d5-cb4eb34e7e26-home" pod "9bf62832-a048-47e7-91d5-cb4eb34e7e26" (UID: "9bf62832-a048-47e7-91d5-cb4eb34e7e26"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:55:53.073836 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:53.073553    4473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bf62832-a048-47e7-91d5-cb4eb34e7e26-model-cache\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:55:53.075920 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:53.075896    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bf62832-a048-47e7-91d5-cb4eb34e7e26-kube-api-access-szqxc" pod "9bf62832-a048-47e7-91d5-cb4eb34e7e26" (UID: "9bf62832-a048-47e7-91d5-cb4eb34e7e26"). InnerVolumeSpecName "kube-api-access-szqxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:55:53.077054 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:53.077032    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bf62832-a048-47e7-91d5-cb4eb34e7e26-tls-certs" pod "9bf62832-a048-47e7-91d5-cb4eb34e7e26" (UID: "9bf62832-a048-47e7-91d5-cb4eb34e7e26"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:55:53.083371 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:53.083338    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bf62832-a048-47e7-91d5-cb4eb34e7e26-dshm" pod "9bf62832-a048-47e7-91d5-cb4eb34e7e26" (UID: "9bf62832-a048-47e7-91d5-cb4eb34e7e26"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:55:53.177518 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:53.175214    4473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bf62832-a048-47e7-91d5-cb4eb34e7e26-tls-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:55:53.177518 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:53.175255    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szqxc\" (UniqueName: \"kubernetes.io/projected/9bf62832-a048-47e7-91d5-cb4eb34e7e26-kube-api-access-szqxc\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:55:53.177518 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:53.175272    4473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bf62832-a048-47e7-91d5-cb4eb34e7e26-tmp-dir\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:55:53.177518 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:53.175289    4473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bf62832-a048-47e7-91d5-cb4eb34e7e26-dshm\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:55:53.177518 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:53.175302    4473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bf62832-a048-47e7-91d5-cb4eb34e7e26-home\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:55:53.496663 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:53.496625    4473 generic.go:356] "Generic (PLEG): container finished" podID="9bf62832-a048-47e7-91d5-cb4eb34e7e26" containerID="72890e92d941f1a7147c0a2b7f5fc300bc442c8200fe08b9b4fe9b5bb464ae24" exitCode=0
Nov 03 09:55:53.496880 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:53.496671    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-627e5f/sub-test-model-627e5f-kserve-6785c9fd7f-z4chr" event={"ID":"9bf62832-a048-47e7-91d5-cb4eb34e7e26","Type":"ContainerDied","Data":"72890e92d941f1a7147c0a2b7f5fc300bc442c8200fe08b9b4fe9b5bb464ae24"}
Nov 03 09:55:53.496880 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:53.496673    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-627e5f/sub-test-model-627e5f-kserve-6785c9fd7f-z4chr"
Nov 03 09:55:53.496880 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:53.496719    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-627e5f/sub-test-model-627e5f-kserve-6785c9fd7f-z4chr" event={"ID":"9bf62832-a048-47e7-91d5-cb4eb34e7e26","Type":"ContainerDied","Data":"576a97e7c68986e302c383853bf4f99cab8ca683dcebabb8939968f0b5cf450f"}
Nov 03 09:55:53.496880 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:53.496738    4473 scope.go:122] "RemoveContainer" containerID="72890e92d941f1a7147c0a2b7f5fc300bc442c8200fe08b9b4fe9b5bb464ae24"
Nov 03 09:55:53.507077 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:53.507054    4473 scope.go:122] "RemoveContainer" containerID="72890e92d941f1a7147c0a2b7f5fc300bc442c8200fe08b9b4fe9b5bb464ae24"
Nov 03 09:55:53.507366 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:55:53.507341    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72890e92d941f1a7147c0a2b7f5fc300bc442c8200fe08b9b4fe9b5bb464ae24\": container with ID starting with 72890e92d941f1a7147c0a2b7f5fc300bc442c8200fe08b9b4fe9b5bb464ae24 not found: ID does not exist" containerID="72890e92d941f1a7147c0a2b7f5fc300bc442c8200fe08b9b4fe9b5bb464ae24"
Nov 03 09:55:53.507458 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:53.507378    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72890e92d941f1a7147c0a2b7f5fc300bc442c8200fe08b9b4fe9b5bb464ae24"} err="failed to get container status \"72890e92d941f1a7147c0a2b7f5fc300bc442c8200fe08b9b4fe9b5bb464ae24\": rpc error: code = NotFound desc = could not find container \"72890e92d941f1a7147c0a2b7f5fc300bc442c8200fe08b9b4fe9b5bb464ae24\": container with ID starting with 72890e92d941f1a7147c0a2b7f5fc300bc442c8200fe08b9b4fe9b5bb464ae24 not found: ID does not exist"
Nov 03 09:55:53.666035 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:53.666006    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-627e5f/sub-test-model-627e5f-kserve-6785c9fd7f-z4chr"]
Nov 03 09:55:53.668663 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:53.668637    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-627e5f/sub-test-model-627e5f-kserve-6785c9fd7f-z4chr"]
Nov 03 09:55:53.878386 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:53.878348    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bf62832-a048-47e7-91d5-cb4eb34e7e26" path="/var/lib/kubelet/pods/9bf62832-a048-47e7-91d5-cb4eb34e7e26/volumes"
Nov 03 09:55:55.568723 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:55.568688    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-b196c8/sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77" podUID="a5dce140-fcd3-420f-bb8e-f0cb8562d047" containerName="main" containerID="cri-o://664e2201cfd2f8afc59ef5820a0cc61f91818fee8d2cebd348b62ff9f6235547" gracePeriod=45
Nov 03 09:55:55.820147 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:55.820090    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b196c8/sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77"
Nov 03 09:55:55.903348 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:55.903315    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5dce140-fcd3-420f-bb8e-f0cb8562d047-home\" (UniqueName: \"kubernetes.io/empty-dir/a5dce140-fcd3-420f-bb8e-f0cb8562d047-home\") pod \"a5dce140-fcd3-420f-bb8e-f0cb8562d047\" (UID: \"a5dce140-fcd3-420f-bb8e-f0cb8562d047\") "
Nov 03 09:55:55.903505 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:55.903368    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5dce140-fcd3-420f-bb8e-f0cb8562d047-kube-api-access-qtwrt\" (UniqueName: \"kubernetes.io/projected/a5dce140-fcd3-420f-bb8e-f0cb8562d047-kube-api-access-qtwrt\") pod \"a5dce140-fcd3-420f-bb8e-f0cb8562d047\" (UID: \"a5dce140-fcd3-420f-bb8e-f0cb8562d047\") "
Nov 03 09:55:55.903505 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:55.903425    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5dce140-fcd3-420f-bb8e-f0cb8562d047-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5dce140-fcd3-420f-bb8e-f0cb8562d047-model-cache\") pod \"a5dce140-fcd3-420f-bb8e-f0cb8562d047\" (UID: \"a5dce140-fcd3-420f-bb8e-f0cb8562d047\") "
Nov 03 09:55:55.903693 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:55.903543    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5dce140-fcd3-420f-bb8e-f0cb8562d047-home" pod "a5dce140-fcd3-420f-bb8e-f0cb8562d047" (UID: "a5dce140-fcd3-420f-bb8e-f0cb8562d047"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:55:55.903693 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:55.903600    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5dce140-fcd3-420f-bb8e-f0cb8562d047-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5dce140-fcd3-420f-bb8e-f0cb8562d047-tmp-dir\") pod \"a5dce140-fcd3-420f-bb8e-f0cb8562d047\" (UID: \"a5dce140-fcd3-420f-bb8e-f0cb8562d047\") "
Nov 03 09:55:55.903693 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:55.903674    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5dce140-fcd3-420f-bb8e-f0cb8562d047-model-cache" pod "a5dce140-fcd3-420f-bb8e-f0cb8562d047" (UID: "a5dce140-fcd3-420f-bb8e-f0cb8562d047"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:55:55.903861 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:55.903696    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5dce140-fcd3-420f-bb8e-f0cb8562d047-tls-certs\" (UniqueName: \"kubernetes.io/secret/a5dce140-fcd3-420f-bb8e-f0cb8562d047-tls-certs\") pod \"a5dce140-fcd3-420f-bb8e-f0cb8562d047\" (UID: \"a5dce140-fcd3-420f-bb8e-f0cb8562d047\") "
Nov 03 09:55:55.903861 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:55.903735    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5dce140-fcd3-420f-bb8e-f0cb8562d047-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5dce140-fcd3-420f-bb8e-f0cb8562d047-dshm\") pod \"a5dce140-fcd3-420f-bb8e-f0cb8562d047\" (UID: \"a5dce140-fcd3-420f-bb8e-f0cb8562d047\") "
Nov 03 09:55:55.903982 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:55.903860    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5dce140-fcd3-420f-bb8e-f0cb8562d047-tmp-dir" pod "a5dce140-fcd3-420f-bb8e-f0cb8562d047" (UID: "a5dce140-fcd3-420f-bb8e-f0cb8562d047"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:55:55.904149 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:55.904131    4473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5dce140-fcd3-420f-bb8e-f0cb8562d047-home\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:55:55.904236 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:55.904156    4473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5dce140-fcd3-420f-bb8e-f0cb8562d047-model-cache\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:55:55.904236 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:55.904171    4473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5dce140-fcd3-420f-bb8e-f0cb8562d047-tmp-dir\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:55:55.905811 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:55.905789    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5dce140-fcd3-420f-bb8e-f0cb8562d047-kube-api-access-qtwrt" pod "a5dce140-fcd3-420f-bb8e-f0cb8562d047" (UID: "a5dce140-fcd3-420f-bb8e-f0cb8562d047"). InnerVolumeSpecName "kube-api-access-qtwrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:55:55.906006 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:55.905987    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5dce140-fcd3-420f-bb8e-f0cb8562d047-dshm" pod "a5dce140-fcd3-420f-bb8e-f0cb8562d047" (UID: "a5dce140-fcd3-420f-bb8e-f0cb8562d047"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:55:55.906078 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:55.906006    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5dce140-fcd3-420f-bb8e-f0cb8562d047-tls-certs" pod "a5dce140-fcd3-420f-bb8e-f0cb8562d047" (UID: "a5dce140-fcd3-420f-bb8e-f0cb8562d047"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:55:56.004747 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:56.004711    4473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5dce140-fcd3-420f-bb8e-f0cb8562d047-tls-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:55:56.004747 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:56.004741    4473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5dce140-fcd3-420f-bb8e-f0cb8562d047-dshm\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:55:56.004976 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:56.004754    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtwrt\" (UniqueName: \"kubernetes.io/projected/a5dce140-fcd3-420f-bb8e-f0cb8562d047-kube-api-access-qtwrt\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:55:56.518179 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:56.518149    4473 generic.go:356] "Generic (PLEG): container finished" podID="a5dce140-fcd3-420f-bb8e-f0cb8562d047" containerID="664e2201cfd2f8afc59ef5820a0cc61f91818fee8d2cebd348b62ff9f6235547" exitCode=0
Nov 03 09:55:56.518370 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:56.518196    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b196c8/sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77"
Nov 03 09:55:56.518370 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:56.518215    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b196c8/sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77" event={"ID":"a5dce140-fcd3-420f-bb8e-f0cb8562d047","Type":"ContainerDied","Data":"664e2201cfd2f8afc59ef5820a0cc61f91818fee8d2cebd348b62ff9f6235547"}
Nov 03 09:55:56.518370 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:56.518247    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b196c8/sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77" event={"ID":"a5dce140-fcd3-420f-bb8e-f0cb8562d047","Type":"ContainerDied","Data":"8c2655b94bd665d95296b2b9a091eb077c3e388f71bcfe71a3a6c85a51fb8516"}
Nov 03 09:55:56.518370 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:56.518264    4473 scope.go:122] "RemoveContainer" containerID="664e2201cfd2f8afc59ef5820a0cc61f91818fee8d2cebd348b62ff9f6235547"
Nov 03 09:55:56.528342 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:56.528326    4473 scope.go:122] "RemoveContainer" containerID="664e2201cfd2f8afc59ef5820a0cc61f91818fee8d2cebd348b62ff9f6235547"
Nov 03 09:55:56.528630 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:55:56.528612    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"664e2201cfd2f8afc59ef5820a0cc61f91818fee8d2cebd348b62ff9f6235547\": container with ID starting with 664e2201cfd2f8afc59ef5820a0cc61f91818fee8d2cebd348b62ff9f6235547 not found: ID does not exist" containerID="664e2201cfd2f8afc59ef5820a0cc61f91818fee8d2cebd348b62ff9f6235547"
Nov 03 09:55:56.528702 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:56.528639    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"664e2201cfd2f8afc59ef5820a0cc61f91818fee8d2cebd348b62ff9f6235547"} err="failed to get container status \"664e2201cfd2f8afc59ef5820a0cc61f91818fee8d2cebd348b62ff9f6235547\": rpc error: code = NotFound desc = could not find container \"664e2201cfd2f8afc59ef5820a0cc61f91818fee8d2cebd348b62ff9f6235547\": container with ID starting with 664e2201cfd2f8afc59ef5820a0cc61f91818fee8d2cebd348b62ff9f6235547 not found: ID does not exist"
Nov 03 09:55:56.686933 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:56.686901    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b196c8/sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77"]
Nov 03 09:55:56.737114 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:56.737079    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b196c8/sub-test-model-b196c8-kserve-6d58c6c8c5-dkw77"]
Nov 03 09:55:57.869264 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:57.869226    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5dce140-fcd3-420f-bb8e-f0cb8562d047" path="/var/lib/kubelet/pods/a5dce140-fcd3-420f-bb8e-f0cb8562d047/volumes"
Nov 03 09:55:58.304718 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:58.304639    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-506e6d-94df8cf47-69862"]
Nov 03 09:55:58.412129 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:55:58.412097    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-506e6d-767cbcb48c-dj25d"]
Nov 03 09:56:02.143509 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:02.143473    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"]
Nov 03 09:56:02.144028 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:02.143685    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd" podUID="909a4fb7-462b-4d1b-a96f-8da66cb52d11" containerName="istio-proxy" containerID="cri-o://a65d2b3382a6f7ce75330c2afcb855315ef5bcbbb37dc8f4976b13a075acab53" gracePeriod=30
Nov 03 09:56:03.305064 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:03.305011    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-506e6d-94df8cf47-69862" podUID="3a1aba26-e2bf-43e8-b373-e3ab815bfd10" containerName="payload-processing" containerID="cri-o://9b8e7279b28ecf8f62deca2c3d0473591b884bb73325e0eff9c930f96a08a343" gracePeriod=25
Nov 03 09:56:03.412869 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:03.412835    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-506e6d-767cbcb48c-dj25d" podUID="f5243dfa-09ba-4203-be20-66b1bdf296f8" containerName="payload-pre-processing" containerID="cri-o://27c5518da647adb7fc3e6510fbb4a2e9f74a6a49b9784291b3077cd28b3f3ea1" gracePeriod=25
Nov 03 09:56:06.052006 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:06.051972    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dee3e12-9twtp"]
Nov 03 09:56:06.101801 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:06.101768    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dee3e12-9twtp"]
Nov 03 09:56:07.074070 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.074028    4473 patch_prober.go:29] interesting pod/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 09:56:07.074458 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.074096    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd" podUID="909a4fb7-462b-4d1b-a96f-8da66cb52d11" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 09:56:07.385144 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.385121    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"
Nov 03 09:56:07.538642 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.538610    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/909a4fb7-462b-4d1b-a96f-8da66cb52d11-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/909a4fb7-462b-4d1b-a96f-8da66cb52d11-credential-socket\") pod \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\" (UID: \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\") "
Nov 03 09:56:07.538818 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.538665    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/909a4fb7-462b-4d1b-a96f-8da66cb52d11-kube-api-access-mzsgn\" (UniqueName: \"kubernetes.io/projected/909a4fb7-462b-4d1b-a96f-8da66cb52d11-kube-api-access-mzsgn\") pod \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\" (UID: \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\") "
Nov 03 09:56:07.538818 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.538697    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/909a4fb7-462b-4d1b-a96f-8da66cb52d11-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/909a4fb7-462b-4d1b-a96f-8da66cb52d11-istio-envoy\") pod \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\" (UID: \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\") "
Nov 03 09:56:07.538818 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.538723    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/909a4fb7-462b-4d1b-a96f-8da66cb52d11-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/909a4fb7-462b-4d1b-a96f-8da66cb52d11-workload-socket\") pod \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\" (UID: \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\") "
Nov 03 09:56:07.538818 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.538792    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/909a4fb7-462b-4d1b-a96f-8da66cb52d11-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/909a4fb7-462b-4d1b-a96f-8da66cb52d11-workload-certs\") pod \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\" (UID: \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\") "
Nov 03 09:56:07.539067 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.538822    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/909a4fb7-462b-4d1b-a96f-8da66cb52d11-istio-token\" (UniqueName: \"kubernetes.io/projected/909a4fb7-462b-4d1b-a96f-8da66cb52d11-istio-token\") pod \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\" (UID: \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\") "
Nov 03 09:56:07.539067 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.538876    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/909a4fb7-462b-4d1b-a96f-8da66cb52d11-credential-socket" pod "909a4fb7-462b-4d1b-a96f-8da66cb52d11" (UID: "909a4fb7-462b-4d1b-a96f-8da66cb52d11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:56:07.539067 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.538934    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/909a4fb7-462b-4d1b-a96f-8da66cb52d11-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/909a4fb7-462b-4d1b-a96f-8da66cb52d11-istiod-ca-cert\") pod \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\" (UID: \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\") "
Nov 03 09:56:07.539230 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.539128    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/909a4fb7-462b-4d1b-a96f-8da66cb52d11-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/909a4fb7-462b-4d1b-a96f-8da66cb52d11-istio-podinfo\") pod \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\" (UID: \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\") "
Nov 03 09:56:07.539230 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.539149    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/909a4fb7-462b-4d1b-a96f-8da66cb52d11-workload-socket" pod "909a4fb7-462b-4d1b-a96f-8da66cb52d11" (UID: "909a4fb7-462b-4d1b-a96f-8da66cb52d11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:56:07.539230 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.539138    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/909a4fb7-462b-4d1b-a96f-8da66cb52d11-workload-certs" pod "909a4fb7-462b-4d1b-a96f-8da66cb52d11" (UID: "909a4fb7-462b-4d1b-a96f-8da66cb52d11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:56:07.539230 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.539213    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/909a4fb7-462b-4d1b-a96f-8da66cb52d11-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/909a4fb7-462b-4d1b-a96f-8da66cb52d11-istio-data\") pod \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\" (UID: \"909a4fb7-462b-4d1b-a96f-8da66cb52d11\") "
Nov 03 09:56:07.539449 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.539314    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/909a4fb7-462b-4d1b-a96f-8da66cb52d11-istiod-ca-cert" pod "909a4fb7-462b-4d1b-a96f-8da66cb52d11" (UID: "909a4fb7-462b-4d1b-a96f-8da66cb52d11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:56:07.539529 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.539500    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/909a4fb7-462b-4d1b-a96f-8da66cb52d11-istio-data" pod "909a4fb7-462b-4d1b-a96f-8da66cb52d11" (UID: "909a4fb7-462b-4d1b-a96f-8da66cb52d11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:56:07.539729 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.539703    4473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/909a4fb7-462b-4d1b-a96f-8da66cb52d11-istio-data\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:56:07.539864 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.539732    4473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/909a4fb7-462b-4d1b-a96f-8da66cb52d11-credential-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:56:07.539864 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.539749    4473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/909a4fb7-462b-4d1b-a96f-8da66cb52d11-workload-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:56:07.539864 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.539763    4473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/909a4fb7-462b-4d1b-a96f-8da66cb52d11-workload-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:56:07.539864 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.539776    4473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/909a4fb7-462b-4d1b-a96f-8da66cb52d11-istiod-ca-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:56:07.541242 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.541217    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/909a4fb7-462b-4d1b-a96f-8da66cb52d11-kube-api-access-mzsgn" pod "909a4fb7-462b-4d1b-a96f-8da66cb52d11" (UID: "909a4fb7-462b-4d1b-a96f-8da66cb52d11"). InnerVolumeSpecName "kube-api-access-mzsgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:56:07.541242 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.541221    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/909a4fb7-462b-4d1b-a96f-8da66cb52d11-istio-envoy" pod "909a4fb7-462b-4d1b-a96f-8da66cb52d11" (UID: "909a4fb7-462b-4d1b-a96f-8da66cb52d11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:56:07.541370 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.541320    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/909a4fb7-462b-4d1b-a96f-8da66cb52d11-istio-token" pod "909a4fb7-462b-4d1b-a96f-8da66cb52d11" (UID: "909a4fb7-462b-4d1b-a96f-8da66cb52d11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:56:07.541468 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.541454    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/909a4fb7-462b-4d1b-a96f-8da66cb52d11-istio-podinfo" pod "909a4fb7-462b-4d1b-a96f-8da66cb52d11" (UID: "909a4fb7-462b-4d1b-a96f-8da66cb52d11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 09:56:07.585181 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.585152    4473 generic.go:356] "Generic (PLEG): container finished" podID="909a4fb7-462b-4d1b-a96f-8da66cb52d11" containerID="a65d2b3382a6f7ce75330c2afcb855315ef5bcbbb37dc8f4976b13a075acab53" exitCode=0
Nov 03 09:56:07.585308 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.585211    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"
Nov 03 09:56:07.585308 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.585243    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd" event={"ID":"909a4fb7-462b-4d1b-a96f-8da66cb52d11","Type":"ContainerDied","Data":"a65d2b3382a6f7ce75330c2afcb855315ef5bcbbb37dc8f4976b13a075acab53"}
Nov 03 09:56:07.585392 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.585309    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd" event={"ID":"909a4fb7-462b-4d1b-a96f-8da66cb52d11","Type":"ContainerDied","Data":"3b9c1e417091725a8d7a558e6d50b97e78ae2b713d93e2e781e725c844b148f6"}
Nov 03 09:56:07.585392 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.585333    4473 scope.go:122] "RemoveContainer" containerID="a65d2b3382a6f7ce75330c2afcb855315ef5bcbbb37dc8f4976b13a075acab53"
Nov 03 09:56:07.595777 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.595760    4473 scope.go:122] "RemoveContainer" containerID="0ecc964583511d973a3acbc4c4e2030887d6841e0d68f884b85cef3bb1e5c8ce"
Nov 03 09:56:07.604800 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.604761    4473 scope.go:122] "RemoveContainer" containerID="a65d2b3382a6f7ce75330c2afcb855315ef5bcbbb37dc8f4976b13a075acab53"
Nov 03 09:56:07.605160 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:56:07.605132    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a65d2b3382a6f7ce75330c2afcb855315ef5bcbbb37dc8f4976b13a075acab53\": container with ID starting with a65d2b3382a6f7ce75330c2afcb855315ef5bcbbb37dc8f4976b13a075acab53 not found: ID does not exist" containerID="a65d2b3382a6f7ce75330c2afcb855315ef5bcbbb37dc8f4976b13a075acab53"
Nov 03 09:56:07.605274 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.605171    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a65d2b3382a6f7ce75330c2afcb855315ef5bcbbb37dc8f4976b13a075acab53"} err="failed to get container status \"a65d2b3382a6f7ce75330c2afcb855315ef5bcbbb37dc8f4976b13a075acab53\": rpc error: code = NotFound desc = could not find container \"a65d2b3382a6f7ce75330c2afcb855315ef5bcbbb37dc8f4976b13a075acab53\": container with ID starting with a65d2b3382a6f7ce75330c2afcb855315ef5bcbbb37dc8f4976b13a075acab53 not found: ID does not exist"
Nov 03 09:56:07.605274 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.605195    4473 scope.go:122] "RemoveContainer" containerID="0ecc964583511d973a3acbc4c4e2030887d6841e0d68f884b85cef3bb1e5c8ce"
Nov 03 09:56:07.605550 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:56:07.605528    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ecc964583511d973a3acbc4c4e2030887d6841e0d68f884b85cef3bb1e5c8ce\": container with ID starting with 0ecc964583511d973a3acbc4c4e2030887d6841e0d68f884b85cef3bb1e5c8ce not found: ID does not exist" containerID="0ecc964583511d973a3acbc4c4e2030887d6841e0d68f884b85cef3bb1e5c8ce"
Nov 03 09:56:07.605638 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.605556    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ecc964583511d973a3acbc4c4e2030887d6841e0d68f884b85cef3bb1e5c8ce"} err="failed to get container status \"0ecc964583511d973a3acbc4c4e2030887d6841e0d68f884b85cef3bb1e5c8ce\": rpc error: code = NotFound desc = could not find container \"0ecc964583511d973a3acbc4c4e2030887d6841e0d68f884b85cef3bb1e5c8ce\": container with ID starting with 0ecc964583511d973a3acbc4c4e2030887d6841e0d68f884b85cef3bb1e5c8ce not found: ID does not exist"
Nov 03 09:56:07.641175 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.641149    4473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/909a4fb7-462b-4d1b-a96f-8da66cb52d11-istio-podinfo\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:56:07.641175 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.641173    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzsgn\" (UniqueName: \"kubernetes.io/projected/909a4fb7-462b-4d1b-a96f-8da66cb52d11-kube-api-access-mzsgn\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:56:07.641335 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.641185    4473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/909a4fb7-462b-4d1b-a96f-8da66cb52d11-istio-envoy\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:56:07.641335 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.641193    4473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/909a4fb7-462b-4d1b-a96f-8da66cb52d11-istio-token\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:56:07.754803 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.754774    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"]
Nov 03 09:56:07.761786 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.761765    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-f9b517-openshift-default-7c7c4484f9-dvlmd"]
Nov 03 09:56:07.869466 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.869436    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0de59ed1-428a-452f-a5ae-eacb005a59d2" path="/var/lib/kubelet/pods/0de59ed1-428a-452f-a5ae-eacb005a59d2/volumes"
Nov 03 09:56:07.869880 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.869866    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="909a4fb7-462b-4d1b-a96f-8da66cb52d11" path="/var/lib/kubelet/pods/909a4fb7-462b-4d1b-a96f-8da66cb52d11/volumes"
Nov 03 09:56:07.875121 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:07.875099    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:56:10.240546 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:10.240508    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-506e6d-767cbcb48c-dj25d" podUID="f5243dfa-09ba-4203-be20-66b1bdf296f8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:56:10.241004 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:10.240559    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-506e6d-94df8cf47-69862" podUID="3a1aba26-e2bf-43e8-b373-e3ab815bfd10" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:56:16.937740 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:56:16.937707    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:56:20.240283 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:20.240233    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-506e6d-94df8cf47-69862" podUID="3a1aba26-e2bf-43e8-b373-e3ab815bfd10" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:56:20.240717 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:20.240306    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-506e6d-767cbcb48c-dj25d" podUID="f5243dfa-09ba-4203-be20-66b1bdf296f8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:56:23.564564 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:23.564498    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-506e6d-94df8cf47-69862"
Nov 03 09:56:23.615246 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:23.615218    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a1aba26-e2bf-43e8-b373-e3ab815bfd10-kube-api-access-vdm2h\" (UniqueName: \"kubernetes.io/projected/3a1aba26-e2bf-43e8-b373-e3ab815bfd10-kube-api-access-vdm2h\") pod \"3a1aba26-e2bf-43e8-b373-e3ab815bfd10\" (UID: \"3a1aba26-e2bf-43e8-b373-e3ab815bfd10\") "
Nov 03 09:56:23.615384 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:23.615344    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a1aba26-e2bf-43e8-b373-e3ab815bfd10-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a1aba26-e2bf-43e8-b373-e3ab815bfd10-plugins-config-volume\") pod \"3a1aba26-e2bf-43e8-b373-e3ab815bfd10\" (UID: \"3a1aba26-e2bf-43e8-b373-e3ab815bfd10\") "
Nov 03 09:56:23.615693 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:23.615668    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a1aba26-e2bf-43e8-b373-e3ab815bfd10-plugins-config-volume" pod "3a1aba26-e2bf-43e8-b373-e3ab815bfd10" (UID: "3a1aba26-e2bf-43e8-b373-e3ab815bfd10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:56:23.617639 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:23.617615    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a1aba26-e2bf-43e8-b373-e3ab815bfd10-kube-api-access-vdm2h" pod "3a1aba26-e2bf-43e8-b373-e3ab815bfd10" (UID: "3a1aba26-e2bf-43e8-b373-e3ab815bfd10"). InnerVolumeSpecName "kube-api-access-vdm2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:56:23.668824 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:23.668802    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-506e6d-767cbcb48c-dj25d"
Nov 03 09:56:23.677630 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:23.677605    4473 generic.go:356] "Generic (PLEG): container finished" podID="f5243dfa-09ba-4203-be20-66b1bdf296f8" containerID="27c5518da647adb7fc3e6510fbb4a2e9f74a6a49b9784291b3077cd28b3f3ea1" exitCode=0
Nov 03 09:56:23.677755 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:23.677648    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-506e6d-767cbcb48c-dj25d"
Nov 03 09:56:23.677755 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:23.677678    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-506e6d-767cbcb48c-dj25d" event={"ID":"f5243dfa-09ba-4203-be20-66b1bdf296f8","Type":"ContainerDied","Data":"27c5518da647adb7fc3e6510fbb4a2e9f74a6a49b9784291b3077cd28b3f3ea1"}
Nov 03 09:56:23.677850 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:23.677767    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-506e6d-767cbcb48c-dj25d" event={"ID":"f5243dfa-09ba-4203-be20-66b1bdf296f8","Type":"ContainerDied","Data":"907f94751125b72fb444fb57884c4bd7b9e19c1512b36162aef4042792bdbab9"}
Nov 03 09:56:23.677850 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:23.677802    4473 scope.go:122] "RemoveContainer" containerID="27c5518da647adb7fc3e6510fbb4a2e9f74a6a49b9784291b3077cd28b3f3ea1"
Nov 03 09:56:23.679095 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:23.679068    4473 generic.go:356] "Generic (PLEG): container finished" podID="3a1aba26-e2bf-43e8-b373-e3ab815bfd10" containerID="9b8e7279b28ecf8f62deca2c3d0473591b884bb73325e0eff9c930f96a08a343" exitCode=0
Nov 03 09:56:23.679194 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:23.679126    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-506e6d-94df8cf47-69862"
Nov 03 09:56:23.679194 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:23.679162    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-506e6d-94df8cf47-69862" event={"ID":"3a1aba26-e2bf-43e8-b373-e3ab815bfd10","Type":"ContainerDied","Data":"9b8e7279b28ecf8f62deca2c3d0473591b884bb73325e0eff9c930f96a08a343"}
Nov 03 09:56:23.679306 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:23.679203    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-506e6d-94df8cf47-69862" event={"ID":"3a1aba26-e2bf-43e8-b373-e3ab815bfd10","Type":"ContainerDied","Data":"b9012cdd5487b86edbf5ac63ec290aafeef62d977b86322bdbe91586593d38db"}
Nov 03 09:56:23.690804 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:23.690787    4473 scope.go:122] "RemoveContainer" containerID="27c5518da647adb7fc3e6510fbb4a2e9f74a6a49b9784291b3077cd28b3f3ea1"
Nov 03 09:56:23.691138 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:56:23.691114    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27c5518da647adb7fc3e6510fbb4a2e9f74a6a49b9784291b3077cd28b3f3ea1\": container with ID starting with 27c5518da647adb7fc3e6510fbb4a2e9f74a6a49b9784291b3077cd28b3f3ea1 not found: ID does not exist" containerID="27c5518da647adb7fc3e6510fbb4a2e9f74a6a49b9784291b3077cd28b3f3ea1"
Nov 03 09:56:23.691257 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:23.691144    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c5518da647adb7fc3e6510fbb4a2e9f74a6a49b9784291b3077cd28b3f3ea1"} err="failed to get container status \"27c5518da647adb7fc3e6510fbb4a2e9f74a6a49b9784291b3077cd28b3f3ea1\": rpc error: code = NotFound desc = could not find container \"27c5518da647adb7fc3e6510fbb4a2e9f74a6a49b9784291b3077cd28b3f3ea1\": container with ID starting with 27c5518da647adb7fc3e6510fbb4a2e9f74a6a49b9784291b3077cd28b3f3ea1 not found: ID does not exist"
Nov 03 09:56:23.691257 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:23.691161    4473 scope.go:122] "RemoveContainer" containerID="9b8e7279b28ecf8f62deca2c3d0473591b884bb73325e0eff9c930f96a08a343"
Nov 03 09:56:23.700201 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:23.700181    4473 scope.go:122] "RemoveContainer" containerID="9b8e7279b28ecf8f62deca2c3d0473591b884bb73325e0eff9c930f96a08a343"
Nov 03 09:56:23.700515 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:56:23.700468    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b8e7279b28ecf8f62deca2c3d0473591b884bb73325e0eff9c930f96a08a343\": container with ID starting with 9b8e7279b28ecf8f62deca2c3d0473591b884bb73325e0eff9c930f96a08a343 not found: ID does not exist" containerID="9b8e7279b28ecf8f62deca2c3d0473591b884bb73325e0eff9c930f96a08a343"
Nov 03 09:56:23.700583 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:23.700526    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b8e7279b28ecf8f62deca2c3d0473591b884bb73325e0eff9c930f96a08a343"} err="failed to get container status \"9b8e7279b28ecf8f62deca2c3d0473591b884bb73325e0eff9c930f96a08a343\": rpc error: code = NotFound desc = could not find container \"9b8e7279b28ecf8f62deca2c3d0473591b884bb73325e0eff9c930f96a08a343\": container with ID starting with 9b8e7279b28ecf8f62deca2c3d0473591b884bb73325e0eff9c930f96a08a343 not found: ID does not exist"
Nov 03 09:56:23.717064 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:23.717043    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5243dfa-09ba-4203-be20-66b1bdf296f8-kube-api-access-x7jx6\" (UniqueName: \"kubernetes.io/projected/f5243dfa-09ba-4203-be20-66b1bdf296f8-kube-api-access-x7jx6\") pod \"f5243dfa-09ba-4203-be20-66b1bdf296f8\" (UID: \"f5243dfa-09ba-4203-be20-66b1bdf296f8\") "
Nov 03 09:56:23.717156 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:23.717126    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5243dfa-09ba-4203-be20-66b1bdf296f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5243dfa-09ba-4203-be20-66b1bdf296f8-plugins-config-volume\") pod \"f5243dfa-09ba-4203-be20-66b1bdf296f8\" (UID: \"f5243dfa-09ba-4203-be20-66b1bdf296f8\") "
Nov 03 09:56:23.717434 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:23.717408    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a1aba26-e2bf-43e8-b373-e3ab815bfd10-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:56:23.717434 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:23.717433    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdm2h\" (UniqueName: \"kubernetes.io/projected/3a1aba26-e2bf-43e8-b373-e3ab815bfd10-kube-api-access-vdm2h\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:56:23.717574 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:23.717530    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5243dfa-09ba-4203-be20-66b1bdf296f8-plugins-config-volume" pod "f5243dfa-09ba-4203-be20-66b1bdf296f8" (UID: "f5243dfa-09ba-4203-be20-66b1bdf296f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:56:23.719164 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:23.719142    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5243dfa-09ba-4203-be20-66b1bdf296f8-kube-api-access-x7jx6" pod "f5243dfa-09ba-4203-be20-66b1bdf296f8" (UID: "f5243dfa-09ba-4203-be20-66b1bdf296f8"). InnerVolumeSpecName "kube-api-access-x7jx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:56:23.818972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:23.818882    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5243dfa-09ba-4203-be20-66b1bdf296f8-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:56:23.818972 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:23.818906    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7jx6\" (UniqueName: \"kubernetes.io/projected/f5243dfa-09ba-4203-be20-66b1bdf296f8-kube-api-access-x7jx6\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:56:23.966408 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:23.966377    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-506e6d-94df8cf47-69862"]
Nov 03 09:56:23.971278 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:23.971253    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-506e6d-94df8cf47-69862"]
Nov 03 09:56:24.095923 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:24.095846    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-506e6d-767cbcb48c-dj25d"]
Nov 03 09:56:24.101241 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:24.101218    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-506e6d-767cbcb48c-dj25d"]
Nov 03 09:56:25.870661 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:25.870627    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a1aba26-e2bf-43e8-b373-e3ab815bfd10" path="/var/lib/kubelet/pods/3a1aba26-e2bf-43e8-b373-e3ab815bfd10/volumes"
Nov 03 09:56:25.871161 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:25.871128    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5243dfa-09ba-4203-be20-66b1bdf296f8" path="/var/lib/kubelet/pods/f5243dfa-09ba-4203-be20-66b1bdf296f8/volumes"
Nov 03 09:56:32.697755 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:32.697718    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:56:34.053149 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:34.053117    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8ee58e4-vtsbx"]
Nov 03 09:56:34.103292 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:34.103263    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8ee58e4-vtsbx"]
Nov 03 09:56:35.484047 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:35.484009    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"]
Nov 03 09:56:35.484591 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:35.484265    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9" podUID="1224423e-4ddf-4902-80c7-19170efb4b38" containerName="istio-proxy" containerID="cri-o://94787c360c9ab3106a6463cff2ac3c84e3825c7197e9ce7100d9609777efae6c" gracePeriod=30
Nov 03 09:56:35.869552 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:35.869522    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2611f42a-9412-437a-8b84-fa09c47a91e6" path="/var/lib/kubelet/pods/2611f42a-9412-437a-8b84-fa09c47a91e6/volumes"
Nov 03 09:56:36.586755 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.586714    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dcd4551-2fzzz"]
Nov 03 09:56:36.587347 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.587327    4473 cpu_manager.go:414] "containerMap: removing container" podUID="c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9" containerName="payload-processing"
Nov 03 09:56:36.587441 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.587350    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9" containerName="payload-processing"
Nov 03 09:56:36.587441 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.587367    4473 cpu_manager.go:414] "containerMap: removing container" podUID="5605651d-4be1-4b2c-a4a3-827f6fd13146" containerName="revoke-keys"
Nov 03 09:56:36.587441 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.587375    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5605651d-4be1-4b2c-a4a3-827f6fd13146" containerName="revoke-keys"
Nov 03 09:56:36.587441 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.587386    4473 cpu_manager.go:414] "containerMap: removing container" podUID="f5243dfa-09ba-4203-be20-66b1bdf296f8" containerName="payload-pre-processing"
Nov 03 09:56:36.587441 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.587394    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5243dfa-09ba-4203-be20-66b1bdf296f8" containerName="payload-pre-processing"
Nov 03 09:56:36.587441 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.587409    4473 cpu_manager.go:414] "containerMap: removing container" podUID="842a3cc8-4062-41df-aa2f-18c19bb342c9" containerName="payload-pre-processing"
Nov 03 09:56:36.587441 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.587416    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="842a3cc8-4062-41df-aa2f-18c19bb342c9" containerName="payload-pre-processing"
Nov 03 09:56:36.587441 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.587431    4473 cpu_manager.go:414] "containerMap: removing container" podUID="a5dce140-fcd3-420f-bb8e-f0cb8562d047" containerName="main"
Nov 03 09:56:36.587441 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.587438    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5dce140-fcd3-420f-bb8e-f0cb8562d047" containerName="main"
Nov 03 09:56:36.587916 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.587458    4473 cpu_manager.go:414] "containerMap: removing container" podUID="909a4fb7-462b-4d1b-a96f-8da66cb52d11" containerName="istio-proxy"
Nov 03 09:56:36.587916 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.587466    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="909a4fb7-462b-4d1b-a96f-8da66cb52d11" containerName="istio-proxy"
Nov 03 09:56:36.587916 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.587480    4473 cpu_manager.go:414] "containerMap: removing container" podUID="909a4fb7-462b-4d1b-a96f-8da66cb52d11" containerName="istio-proxy"
Nov 03 09:56:36.587916 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.587488    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="909a4fb7-462b-4d1b-a96f-8da66cb52d11" containerName="istio-proxy"
Nov 03 09:56:36.587916 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.587523    4473 cpu_manager.go:414] "containerMap: removing container" podUID="9bf62832-a048-47e7-91d5-cb4eb34e7e26" containerName="main"
Nov 03 09:56:36.587916 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.587531    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bf62832-a048-47e7-91d5-cb4eb34e7e26" containerName="main"
Nov 03 09:56:36.587916 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.587551    4473 cpu_manager.go:414] "containerMap: removing container" podUID="3a1aba26-e2bf-43e8-b373-e3ab815bfd10" containerName="payload-processing"
Nov 03 09:56:36.587916 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.587559    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a1aba26-e2bf-43e8-b373-e3ab815bfd10" containerName="payload-processing"
Nov 03 09:56:36.587916 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.587655    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="909a4fb7-462b-4d1b-a96f-8da66cb52d11" containerName="istio-proxy"
Nov 03 09:56:36.587916 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.587675    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bf62832-a048-47e7-91d5-cb4eb34e7e26" containerName="main"
Nov 03 09:56:36.587916 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.587686    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a1aba26-e2bf-43e8-b373-e3ab815bfd10" containerName="payload-processing"
Nov 03 09:56:36.587916 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.587697    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="c90b6d7d-56d5-4a0b-a4ac-295f1d36fce9" containerName="payload-processing"
Nov 03 09:56:36.587916 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.587706    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="5605651d-4be1-4b2c-a4a3-827f6fd13146" containerName="revoke-keys"
Nov 03 09:56:36.587916 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.587716    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5243dfa-09ba-4203-be20-66b1bdf296f8" containerName="payload-pre-processing"
Nov 03 09:56:36.587916 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.587726    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="842a3cc8-4062-41df-aa2f-18c19bb342c9" containerName="payload-pre-processing"
Nov 03 09:56:36.587916 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.587736    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5dce140-fcd3-420f-bb8e-f0cb8562d047" containerName="main"
Nov 03 09:56:36.592715 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.592684    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dcd4551-2fzzz"
Nov 03 09:56:36.647343 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.647321    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dcd4551-2fzzz"]
Nov 03 09:56:36.694304 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.694282    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:56:36.694449 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.694315    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:56:36.694449 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.694363    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:56:36.696566 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.696554    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:56:36.759865 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.759834    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61529bbf-0a40-447b-b166-5bb4e6836cce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4dcd4551-2fzzz\" (UID: \"61529bbf-0a40-447b-b166-5bb4e6836cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dcd4551-2fzzz"
Nov 03 09:56:36.760095 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.760071    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzkt8\" (UniqueName: \"kubernetes.io/projected/61529bbf-0a40-447b-b166-5bb4e6836cce-kube-api-access-xzkt8\") pod \"maas-api-revoke-sub-4dcd4551-2fzzz\" (UID: \"61529bbf-0a40-447b-b166-5bb4e6836cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dcd4551-2fzzz"
Nov 03 09:56:36.760284 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.760270    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61529bbf-0a40-447b-b166-5bb4e6836cce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4dcd4551-2fzzz\" (UID: \"61529bbf-0a40-447b-b166-5bb4e6836cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dcd4551-2fzzz"
Nov 03 09:56:36.861457 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.861382    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61529bbf-0a40-447b-b166-5bb4e6836cce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4dcd4551-2fzzz\" (UID: \"61529bbf-0a40-447b-b166-5bb4e6836cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dcd4551-2fzzz"
Nov 03 09:56:36.861457 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.861412    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzkt8\" (UniqueName: \"kubernetes.io/projected/61529bbf-0a40-447b-b166-5bb4e6836cce-kube-api-access-xzkt8\") pod \"maas-api-revoke-sub-4dcd4551-2fzzz\" (UID: \"61529bbf-0a40-447b-b166-5bb4e6836cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dcd4551-2fzzz"
Nov 03 09:56:36.861701 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.861466    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61529bbf-0a40-447b-b166-5bb4e6836cce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4dcd4551-2fzzz\" (UID: \"61529bbf-0a40-447b-b166-5bb4e6836cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dcd4551-2fzzz"
Nov 03 09:56:36.862163 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.862141    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61529bbf-0a40-447b-b166-5bb4e6836cce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4dcd4551-2fzzz\" (UID: \"61529bbf-0a40-447b-b166-5bb4e6836cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dcd4551-2fzzz"
Nov 03 09:56:36.863929 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.863908    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61529bbf-0a40-447b-b166-5bb4e6836cce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4dcd4551-2fzzz\" (UID: \"61529bbf-0a40-447b-b166-5bb4e6836cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dcd4551-2fzzz"
Nov 03 09:56:36.920607 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:36.920573    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzkt8\" (UniqueName: \"kubernetes.io/projected/61529bbf-0a40-447b-b166-5bb4e6836cce-kube-api-access-xzkt8\") pod \"maas-api-revoke-sub-4dcd4551-2fzzz\" (UID: \"61529bbf-0a40-447b-b166-5bb4e6836cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dcd4551-2fzzz"
Nov 03 09:56:37.205075 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:37.205045    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dcd4551-2fzzz"
Nov 03 09:56:37.579745 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:37.579717    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dcd4551-2fzzz"]
Nov 03 09:56:37.580921 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:56:37.580894    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61529bbf_0a40_447b_b166_5bb4e6836cce.slice/crio-ce8007d087414fcedc8e19a447e0ab2ad5767b9e4f60e837316746ddcdce9828 WatchSource:0}: Error finding container ce8007d087414fcedc8e19a447e0ab2ad5767b9e4f60e837316746ddcdce9828: Status 404 returned error can't find the container with id ce8007d087414fcedc8e19a447e0ab2ad5767b9e4f60e837316746ddcdce9828
Nov 03 09:56:37.771096 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:37.771066    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dcd4551-2fzzz" event={"ID":"61529bbf-0a40-447b-b166-5bb4e6836cce","Type":"ContainerStarted","Data":"4a669c56d60c2e7d7a7ffd24c9d64f4e03d398d318286906facf227ecc8e3b4f"}
Nov 03 09:56:37.771444 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:37.771127    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dcd4551-2fzzz" event={"ID":"61529bbf-0a40-447b-b166-5bb4e6836cce","Type":"ContainerStarted","Data":"ce8007d087414fcedc8e19a447e0ab2ad5767b9e4f60e837316746ddcdce9828"}
Nov 03 09:56:37.888369 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:37.888317    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dcd4551-2fzzz" podStartSLOduration=1.88830535 podStartE2EDuration="1.88830535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:56:37.887871988 +0000 UTC m=+4622.156935441" watchObservedRunningTime="2026-11-03 09:56:37.88830535 +0000 UTC m=+4622.157368783"
Nov 03 09:56:38.778847 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:38.778763    4473 generic.go:356] "Generic (PLEG): container finished" podID="61529bbf-0a40-447b-b166-5bb4e6836cce" containerID="4a669c56d60c2e7d7a7ffd24c9d64f4e03d398d318286906facf227ecc8e3b4f" exitCode=0
Nov 03 09:56:38.779249 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:38.778847    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dcd4551-2fzzz" event={"ID":"61529bbf-0a40-447b-b166-5bb4e6836cce","Type":"ContainerDied","Data":"4a669c56d60c2e7d7a7ffd24c9d64f4e03d398d318286906facf227ecc8e3b4f"}
Nov 03 09:56:39.928871 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:39.928844    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dcd4551-2fzzz"
Nov 03 09:56:39.971007 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:39.970940    4473 patch_prober.go:29] interesting pod/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 09:56:39.971187 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:39.971029    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9" podUID="1224423e-4ddf-4902-80c7-19170efb4b38" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 09:56:40.110820 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.097483    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61529bbf-0a40-447b-b166-5bb4e6836cce-kube-api-access-xzkt8\" (UniqueName: \"kubernetes.io/projected/61529bbf-0a40-447b-b166-5bb4e6836cce-kube-api-access-xzkt8\") pod \"61529bbf-0a40-447b-b166-5bb4e6836cce\" (UID: \"61529bbf-0a40-447b-b166-5bb4e6836cce\") "
Nov 03 09:56:40.110820 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.097683    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61529bbf-0a40-447b-b166-5bb4e6836cce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61529bbf-0a40-447b-b166-5bb4e6836cce-maas-api-openshift-ca\") pod \"61529bbf-0a40-447b-b166-5bb4e6836cce\" (UID: \"61529bbf-0a40-447b-b166-5bb4e6836cce\") "
Nov 03 09:56:40.110820 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.097739    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61529bbf-0a40-447b-b166-5bb4e6836cce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61529bbf-0a40-447b-b166-5bb4e6836cce-maas-api-xks-ca\") pod \"61529bbf-0a40-447b-b166-5bb4e6836cce\" (UID: \"61529bbf-0a40-447b-b166-5bb4e6836cce\") "
Nov 03 09:56:40.111600 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.111552    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61529bbf-0a40-447b-b166-5bb4e6836cce-maas-api-openshift-ca" pod "61529bbf-0a40-447b-b166-5bb4e6836cce" (UID: "61529bbf-0a40-447b-b166-5bb4e6836cce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:56:40.117703 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.117629    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61529bbf-0a40-447b-b166-5bb4e6836cce-maas-api-xks-ca" pod "61529bbf-0a40-447b-b166-5bb4e6836cce" (UID: "61529bbf-0a40-447b-b166-5bb4e6836cce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:56:40.119340 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.119308    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61529bbf-0a40-447b-b166-5bb4e6836cce-kube-api-access-xzkt8" pod "61529bbf-0a40-447b-b166-5bb4e6836cce" (UID: "61529bbf-0a40-447b-b166-5bb4e6836cce"). InnerVolumeSpecName "kube-api-access-xzkt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:56:40.199570 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.199533    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61529bbf-0a40-447b-b166-5bb4e6836cce-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:56:40.199570 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.199565    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61529bbf-0a40-447b-b166-5bb4e6836cce-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:56:40.199570 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.199575    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzkt8\" (UniqueName: \"kubernetes.io/projected/61529bbf-0a40-447b-b166-5bb4e6836cce-kube-api-access-xzkt8\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:56:40.731508 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.731482    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"
Nov 03 09:56:40.791853 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.791786    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dcd4551-2fzzz"
Nov 03 09:56:40.791853 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.791824    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dcd4551-2fzzz" event={"ID":"61529bbf-0a40-447b-b166-5bb4e6836cce","Type":"ContainerDied","Data":"ce8007d087414fcedc8e19a447e0ab2ad5767b9e4f60e837316746ddcdce9828"}
Nov 03 09:56:40.792067 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.791869    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce8007d087414fcedc8e19a447e0ab2ad5767b9e4f60e837316746ddcdce9828"
Nov 03 09:56:40.793522 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.793493    4473 generic.go:356] "Generic (PLEG): container finished" podID="1224423e-4ddf-4902-80c7-19170efb4b38" containerID="94787c360c9ab3106a6463cff2ac3c84e3825c7197e9ce7100d9609777efae6c" exitCode=0
Nov 03 09:56:40.793661 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.793539    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"
Nov 03 09:56:40.793661 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.793570    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9" event={"ID":"1224423e-4ddf-4902-80c7-19170efb4b38","Type":"ContainerDied","Data":"94787c360c9ab3106a6463cff2ac3c84e3825c7197e9ce7100d9609777efae6c"}
Nov 03 09:56:40.793661 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.793617    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9" event={"ID":"1224423e-4ddf-4902-80c7-19170efb4b38","Type":"ContainerDied","Data":"cffb13f97c16b3cfdf7df6994d1bc56fc86326cc4f8742d062368b485a824cbb"}
Nov 03 09:56:40.793661 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.793640    4473 scope.go:122] "RemoveContainer" containerID="94787c360c9ab3106a6463cff2ac3c84e3825c7197e9ce7100d9609777efae6c"
Nov 03 09:56:40.803717 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.803700    4473 scope.go:122] "RemoveContainer" containerID="bf7662cc3fa8000a9e3178a6454d3618d2612e783aa508c9c80c1dd8b0a01f27"
Nov 03 09:56:40.812457 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.812436    4473 scope.go:122] "RemoveContainer" containerID="94787c360c9ab3106a6463cff2ac3c84e3825c7197e9ce7100d9609777efae6c"
Nov 03 09:56:40.812701 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:56:40.812677    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94787c360c9ab3106a6463cff2ac3c84e3825c7197e9ce7100d9609777efae6c\": container with ID starting with 94787c360c9ab3106a6463cff2ac3c84e3825c7197e9ce7100d9609777efae6c not found: ID does not exist" containerID="94787c360c9ab3106a6463cff2ac3c84e3825c7197e9ce7100d9609777efae6c"
Nov 03 09:56:40.812761 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.812711    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94787c360c9ab3106a6463cff2ac3c84e3825c7197e9ce7100d9609777efae6c"} err="failed to get container status \"94787c360c9ab3106a6463cff2ac3c84e3825c7197e9ce7100d9609777efae6c\": rpc error: code = NotFound desc = could not find container \"94787c360c9ab3106a6463cff2ac3c84e3825c7197e9ce7100d9609777efae6c\": container with ID starting with 94787c360c9ab3106a6463cff2ac3c84e3825c7197e9ce7100d9609777efae6c not found: ID does not exist"
Nov 03 09:56:40.812761 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.812729    4473 scope.go:122] "RemoveContainer" containerID="bf7662cc3fa8000a9e3178a6454d3618d2612e783aa508c9c80c1dd8b0a01f27"
Nov 03 09:56:40.812961 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:56:40.812926    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf7662cc3fa8000a9e3178a6454d3618d2612e783aa508c9c80c1dd8b0a01f27\": container with ID starting with bf7662cc3fa8000a9e3178a6454d3618d2612e783aa508c9c80c1dd8b0a01f27 not found: ID does not exist" containerID="bf7662cc3fa8000a9e3178a6454d3618d2612e783aa508c9c80c1dd8b0a01f27"
Nov 03 09:56:40.813009 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.812961    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf7662cc3fa8000a9e3178a6454d3618d2612e783aa508c9c80c1dd8b0a01f27"} err="failed to get container status \"bf7662cc3fa8000a9e3178a6454d3618d2612e783aa508c9c80c1dd8b0a01f27\": rpc error: code = NotFound desc = could not find container \"bf7662cc3fa8000a9e3178a6454d3618d2612e783aa508c9c80c1dd8b0a01f27\": container with ID starting with bf7662cc3fa8000a9e3178a6454d3618d2612e783aa508c9c80c1dd8b0a01f27 not found: ID does not exist"
Nov 03 09:56:40.907273 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.907244    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1224423e-4ddf-4902-80c7-19170efb4b38-istio-token\" (UniqueName: \"kubernetes.io/projected/1224423e-4ddf-4902-80c7-19170efb4b38-istio-token\") pod \"1224423e-4ddf-4902-80c7-19170efb4b38\" (UID: \"1224423e-4ddf-4902-80c7-19170efb4b38\") "
Nov 03 09:56:40.907487 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.907311    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1224423e-4ddf-4902-80c7-19170efb4b38-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1224423e-4ddf-4902-80c7-19170efb4b38-workload-certs\") pod \"1224423e-4ddf-4902-80c7-19170efb4b38\" (UID: \"1224423e-4ddf-4902-80c7-19170efb4b38\") "
Nov 03 09:56:40.907487 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.907388    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1224423e-4ddf-4902-80c7-19170efb4b38-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1224423e-4ddf-4902-80c7-19170efb4b38-istiod-ca-cert\") pod \"1224423e-4ddf-4902-80c7-19170efb4b38\" (UID: \"1224423e-4ddf-4902-80c7-19170efb4b38\") "
Nov 03 09:56:40.907487 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.907462    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1224423e-4ddf-4902-80c7-19170efb4b38-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1224423e-4ddf-4902-80c7-19170efb4b38-istio-envoy\") pod \"1224423e-4ddf-4902-80c7-19170efb4b38\" (UID: \"1224423e-4ddf-4902-80c7-19170efb4b38\") "
Nov 03 09:56:40.907891 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.907509    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1224423e-4ddf-4902-80c7-19170efb4b38-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1224423e-4ddf-4902-80c7-19170efb4b38-credential-socket\") pod \"1224423e-4ddf-4902-80c7-19170efb4b38\" (UID: \"1224423e-4ddf-4902-80c7-19170efb4b38\") "
Nov 03 09:56:40.907891 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.907558    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1224423e-4ddf-4902-80c7-19170efb4b38-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1224423e-4ddf-4902-80c7-19170efb4b38-istio-podinfo\") pod \"1224423e-4ddf-4902-80c7-19170efb4b38\" (UID: \"1224423e-4ddf-4902-80c7-19170efb4b38\") "
Nov 03 09:56:40.907891 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.907590    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1224423e-4ddf-4902-80c7-19170efb4b38-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1224423e-4ddf-4902-80c7-19170efb4b38-workload-socket\") pod \"1224423e-4ddf-4902-80c7-19170efb4b38\" (UID: \"1224423e-4ddf-4902-80c7-19170efb4b38\") "
Nov 03 09:56:40.907891 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.907632    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1224423e-4ddf-4902-80c7-19170efb4b38-kube-api-access-7z6bz\" (UniqueName: \"kubernetes.io/projected/1224423e-4ddf-4902-80c7-19170efb4b38-kube-api-access-7z6bz\") pod \"1224423e-4ddf-4902-80c7-19170efb4b38\" (UID: \"1224423e-4ddf-4902-80c7-19170efb4b38\") "
Nov 03 09:56:40.907891 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.907787    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1224423e-4ddf-4902-80c7-19170efb4b38-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1224423e-4ddf-4902-80c7-19170efb4b38-istio-data\") pod \"1224423e-4ddf-4902-80c7-19170efb4b38\" (UID: \"1224423e-4ddf-4902-80c7-19170efb4b38\") "
Nov 03 09:56:40.908277 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.908225    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1224423e-4ddf-4902-80c7-19170efb4b38-istio-data" pod "1224423e-4ddf-4902-80c7-19170efb4b38" (UID: "1224423e-4ddf-4902-80c7-19170efb4b38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:56:40.908498 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.908443    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1224423e-4ddf-4902-80c7-19170efb4b38-credential-socket" pod "1224423e-4ddf-4902-80c7-19170efb4b38" (UID: "1224423e-4ddf-4902-80c7-19170efb4b38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:56:40.908990 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.908690    4473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1224423e-4ddf-4902-80c7-19170efb4b38-credential-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:56:40.908990 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.908722    4473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1224423e-4ddf-4902-80c7-19170efb4b38-istio-data\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:56:40.909189 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.909172    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1224423e-4ddf-4902-80c7-19170efb4b38-workload-certs" pod "1224423e-4ddf-4902-80c7-19170efb4b38" (UID: "1224423e-4ddf-4902-80c7-19170efb4b38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:56:40.909885 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.909865    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1224423e-4ddf-4902-80c7-19170efb4b38-istiod-ca-cert" pod "1224423e-4ddf-4902-80c7-19170efb4b38" (UID: "1224423e-4ddf-4902-80c7-19170efb4b38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:56:40.912975 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.911417    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1224423e-4ddf-4902-80c7-19170efb4b38-kube-api-access-7z6bz" pod "1224423e-4ddf-4902-80c7-19170efb4b38" (UID: "1224423e-4ddf-4902-80c7-19170efb4b38"). InnerVolumeSpecName "kube-api-access-7z6bz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:56:40.912975 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.912115    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1224423e-4ddf-4902-80c7-19170efb4b38-istio-token" pod "1224423e-4ddf-4902-80c7-19170efb4b38" (UID: "1224423e-4ddf-4902-80c7-19170efb4b38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:56:40.912975 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.912198    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1224423e-4ddf-4902-80c7-19170efb4b38-istio-podinfo" pod "1224423e-4ddf-4902-80c7-19170efb4b38" (UID: "1224423e-4ddf-4902-80c7-19170efb4b38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 09:56:40.912975 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.912831    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd"]
Nov 03 09:56:40.913793 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.913689    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1224423e-4ddf-4902-80c7-19170efb4b38-workload-socket" pod "1224423e-4ddf-4902-80c7-19170efb4b38" (UID: "1224423e-4ddf-4902-80c7-19170efb4b38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:56:40.914350 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:40.914326    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1224423e-4ddf-4902-80c7-19170efb4b38-istio-envoy" pod "1224423e-4ddf-4902-80c7-19170efb4b38" (UID: "1224423e-4ddf-4902-80c7-19170efb4b38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:56:41.009496 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:41.009459    4473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1224423e-4ddf-4902-80c7-19170efb4b38-istio-token\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:56:41.009496 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:41.009485    4473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1224423e-4ddf-4902-80c7-19170efb4b38-workload-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:56:41.009496 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:41.009497    4473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1224423e-4ddf-4902-80c7-19170efb4b38-istiod-ca-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:56:41.009496 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:41.009505    4473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1224423e-4ddf-4902-80c7-19170efb4b38-istio-envoy\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:56:41.010013 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:41.009514    4473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1224423e-4ddf-4902-80c7-19170efb4b38-istio-podinfo\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:56:41.010013 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:41.009522    4473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1224423e-4ddf-4902-80c7-19170efb4b38-workload-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:56:41.010013 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:41.009530    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z6bz\" (UniqueName: \"kubernetes.io/projected/1224423e-4ddf-4902-80c7-19170efb4b38-kube-api-access-7z6bz\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:56:41.220179 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:41.220134    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"]
Nov 03 09:56:41.222479 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:41.222457    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-506e6d-openshift-default-7f7f949fb5-5t2b9"]
Nov 03 09:56:41.869958 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:41.869920    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1224423e-4ddf-4902-80c7-19170efb4b38" path="/var/lib/kubelet/pods/1224423e-4ddf-4902-80c7-19170efb4b38/volumes"
Nov 03 09:56:45.059135 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:45.059099    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c77af362-hqs7b"]
Nov 03 09:56:45.109613 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:45.109581    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c77af362-hqs7b"]
Nov 03 09:56:45.868912 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:45.868879    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7" path="/var/lib/kubelet/pods/6b50ede6-3ef0-400f-a3ca-c014a8e0f7b7/volumes"
Nov 03 09:56:46.363432 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:46.363355    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71076da4-bnqjf"]
Nov 03 09:56:46.363989 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:46.363971    4473 cpu_manager.go:414] "containerMap: removing container" podUID="1224423e-4ddf-4902-80c7-19170efb4b38" containerName="istio-proxy"
Nov 03 09:56:46.363989 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:46.363989    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1224423e-4ddf-4902-80c7-19170efb4b38" containerName="istio-proxy"
Nov 03 09:56:46.364100 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:46.363999    4473 cpu_manager.go:414] "containerMap: removing container" podUID="1224423e-4ddf-4902-80c7-19170efb4b38" containerName="istio-proxy"
Nov 03 09:56:46.364100 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:46.364004    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1224423e-4ddf-4902-80c7-19170efb4b38" containerName="istio-proxy"
Nov 03 09:56:46.364100 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:46.364013    4473 cpu_manager.go:414] "containerMap: removing container" podUID="61529bbf-0a40-447b-b166-5bb4e6836cce" containerName="revoke-keys"
Nov 03 09:56:46.364100 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:46.364019    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61529bbf-0a40-447b-b166-5bb4e6836cce" containerName="revoke-keys"
Nov 03 09:56:46.364254 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:46.364113    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="909a4fb7-462b-4d1b-a96f-8da66cb52d11" containerName="istio-proxy"
Nov 03 09:56:46.364254 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:46.364122    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="1224423e-4ddf-4902-80c7-19170efb4b38" containerName="istio-proxy"
Nov 03 09:56:46.364254 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:46.364128    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="1224423e-4ddf-4902-80c7-19170efb4b38" containerName="istio-proxy"
Nov 03 09:56:46.364254 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:46.364134    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="61529bbf-0a40-447b-b166-5bb4e6836cce" containerName="revoke-keys"
Nov 03 09:56:46.368625 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:46.368611    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71076da4-bnqjf"
Nov 03 09:56:46.426149 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:46.426106    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71076da4-bnqjf"]
Nov 03 09:56:46.472553 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:46.472522    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:56:46.472765 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:46.472594    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:56:46.472765 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:46.472625    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:56:46.472765 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:46.472644    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:56:46.565380 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:46.565346    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c91a81-8f4f-4161-9189-01d948f1b55c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71076da4-bnqjf\" (UID: \"a7c91a81-8f4f-4161-9189-01d948f1b55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71076da4-bnqjf"
Nov 03 09:56:46.565590 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:46.565394    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjhs4\" (UniqueName: \"kubernetes.io/projected/a7c91a81-8f4f-4161-9189-01d948f1b55c-kube-api-access-xjhs4\") pod \"maas-api-revoke-sub-71076da4-bnqjf\" (UID: \"a7c91a81-8f4f-4161-9189-01d948f1b55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71076da4-bnqjf"
Nov 03 09:56:46.565590 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:46.565520    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c91a81-8f4f-4161-9189-01d948f1b55c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71076da4-bnqjf\" (UID: \"a7c91a81-8f4f-4161-9189-01d948f1b55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71076da4-bnqjf"
Nov 03 09:56:46.666499 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:46.666464    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c91a81-8f4f-4161-9189-01d948f1b55c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71076da4-bnqjf\" (UID: \"a7c91a81-8f4f-4161-9189-01d948f1b55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71076da4-bnqjf"
Nov 03 09:56:46.666648 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:46.666550    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c91a81-8f4f-4161-9189-01d948f1b55c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71076da4-bnqjf\" (UID: \"a7c91a81-8f4f-4161-9189-01d948f1b55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71076da4-bnqjf"
Nov 03 09:56:46.666648 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:46.666583    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjhs4\" (UniqueName: \"kubernetes.io/projected/a7c91a81-8f4f-4161-9189-01d948f1b55c-kube-api-access-xjhs4\") pod \"maas-api-revoke-sub-71076da4-bnqjf\" (UID: \"a7c91a81-8f4f-4161-9189-01d948f1b55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71076da4-bnqjf"
Nov 03 09:56:46.667315 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:46.667238    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c91a81-8f4f-4161-9189-01d948f1b55c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71076da4-bnqjf\" (UID: \"a7c91a81-8f4f-4161-9189-01d948f1b55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71076da4-bnqjf"
Nov 03 09:56:46.669031 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:46.669002    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c91a81-8f4f-4161-9189-01d948f1b55c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71076da4-bnqjf\" (UID: \"a7c91a81-8f4f-4161-9189-01d948f1b55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71076da4-bnqjf"
Nov 03 09:56:46.726175 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:46.726140    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjhs4\" (UniqueName: \"kubernetes.io/projected/a7c91a81-8f4f-4161-9189-01d948f1b55c-kube-api-access-xjhs4\") pod \"maas-api-revoke-sub-71076da4-bnqjf\" (UID: \"a7c91a81-8f4f-4161-9189-01d948f1b55c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71076da4-bnqjf"
Nov 03 09:56:46.936765 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:56:46.936635    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:56:46.981558 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:46.981527    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71076da4-bnqjf"
Nov 03 09:56:47.358501 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:47.358461    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71076da4-bnqjf"]
Nov 03 09:56:47.359771 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:56:47.359742    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7c91a81_8f4f_4161_9189_01d948f1b55c.slice/crio-13a1fe165c7964440f1c9651821006b1108c5b9cbd501f54626adc280e9b0d5f WatchSource:0}: Error finding container 13a1fe165c7964440f1c9651821006b1108c5b9cbd501f54626adc280e9b0d5f: Status 404 returned error can't find the container with id 13a1fe165c7964440f1c9651821006b1108c5b9cbd501f54626adc280e9b0d5f
Nov 03 09:56:47.836376 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:47.836327    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71076da4-bnqjf" event={"ID":"a7c91a81-8f4f-4161-9189-01d948f1b55c","Type":"ContainerStarted","Data":"de44b78155fa522d5d3676f048b7124cb0506509c0dfedeb8bee7da7f0babb4d"}
Nov 03 09:56:47.836757 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:47.836399    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71076da4-bnqjf" event={"ID":"a7c91a81-8f4f-4161-9189-01d948f1b55c","Type":"ContainerStarted","Data":"13a1fe165c7964440f1c9651821006b1108c5b9cbd501f54626adc280e9b0d5f"}
Nov 03 09:56:47.956717 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:47.956616    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71076da4-bnqjf" podStartSLOduration=1.9566030589999999 podStartE2EDuration="1.956603059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 09:56:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 09:56:47.953889155 +0000 UTC m=+4632.222952609" watchObservedRunningTime="2026-11-03 09:56:47.956603059 +0000 UTC m=+4632.225666493"
Nov 03 09:56:48.843319 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:48.843288    4473 generic.go:356] "Generic (PLEG): container finished" podID="a7c91a81-8f4f-4161-9189-01d948f1b55c" containerID="de44b78155fa522d5d3676f048b7124cb0506509c0dfedeb8bee7da7f0babb4d" exitCode=0
Nov 03 09:56:48.843732 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:48.843405    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71076da4-bnqjf" event={"ID":"a7c91a81-8f4f-4161-9189-01d948f1b55c","Type":"ContainerDied","Data":"de44b78155fa522d5d3676f048b7124cb0506509c0dfedeb8bee7da7f0babb4d"}
Nov 03 09:56:50.003167 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:50.003132    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71076da4-bnqjf"
Nov 03 09:56:50.106714 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:50.104370    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7c91a81-8f4f-4161-9189-01d948f1b55c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c91a81-8f4f-4161-9189-01d948f1b55c-maas-api-openshift-ca\") pod \"a7c91a81-8f4f-4161-9189-01d948f1b55c\" (UID: \"a7c91a81-8f4f-4161-9189-01d948f1b55c\") "
Nov 03 09:56:50.106714 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:50.104428    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7c91a81-8f4f-4161-9189-01d948f1b55c-kube-api-access-xjhs4\" (UniqueName: \"kubernetes.io/projected/a7c91a81-8f4f-4161-9189-01d948f1b55c-kube-api-access-xjhs4\") pod \"a7c91a81-8f4f-4161-9189-01d948f1b55c\" (UID: \"a7c91a81-8f4f-4161-9189-01d948f1b55c\") "
Nov 03 09:56:50.106714 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:50.104505    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7c91a81-8f4f-4161-9189-01d948f1b55c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c91a81-8f4f-4161-9189-01d948f1b55c-maas-api-xks-ca\") pod \"a7c91a81-8f4f-4161-9189-01d948f1b55c\" (UID: \"a7c91a81-8f4f-4161-9189-01d948f1b55c\") "
Nov 03 09:56:50.118285 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:50.118249    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7c91a81-8f4f-4161-9189-01d948f1b55c-maas-api-openshift-ca" pod "a7c91a81-8f4f-4161-9189-01d948f1b55c" (UID: "a7c91a81-8f4f-4161-9189-01d948f1b55c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:56:50.128434 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:50.128398    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7c91a81-8f4f-4161-9189-01d948f1b55c-maas-api-xks-ca" pod "a7c91a81-8f4f-4161-9189-01d948f1b55c" (UID: "a7c91a81-8f4f-4161-9189-01d948f1b55c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:56:50.129673 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:50.129640    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c91a81-8f4f-4161-9189-01d948f1b55c-kube-api-access-xjhs4" pod "a7c91a81-8f4f-4161-9189-01d948f1b55c" (UID: "a7c91a81-8f4f-4161-9189-01d948f1b55c"). InnerVolumeSpecName "kube-api-access-xjhs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:56:50.206262 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:50.206229    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c91a81-8f4f-4161-9189-01d948f1b55c-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:56:50.206262 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:50.206258    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjhs4\" (UniqueName: \"kubernetes.io/projected/a7c91a81-8f4f-4161-9189-01d948f1b55c-kube-api-access-xjhs4\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:56:50.206262 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:50.206267    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c91a81-8f4f-4161-9189-01d948f1b55c-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:56:50.721823 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:50.721790    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj"]
Nov 03 09:56:50.771347 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:50.771314    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn"]
Nov 03 09:56:50.857921 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:50.857896    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71076da4-bnqjf"
Nov 03 09:56:50.858129 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:50.857888    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71076da4-bnqjf" event={"ID":"a7c91a81-8f4f-4161-9189-01d948f1b55c","Type":"ContainerDied","Data":"13a1fe165c7964440f1c9651821006b1108c5b9cbd501f54626adc280e9b0d5f"}
Nov 03 09:56:50.858129 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:50.858009    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13a1fe165c7964440f1c9651821006b1108c5b9cbd501f54626adc280e9b0d5f"
Nov 03 09:56:51.641753 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:51.641721    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:56:55.934455 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:55.934408    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd" podUID="cf4cf2fc-612a-4f5c-a596-543036dde7e3" containerName="main" containerID="cri-o://1a185f8717c5d7f7d5f232ad5e2c83487feb26ce7edb70a70b31dcbe11f71cfd" gracePeriod=45
Nov 03 09:56:56.181628 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:56.181605    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd"
Nov 03 09:56:56.265133 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:56.265044    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf4cf2fc-612a-4f5c-a596-543036dde7e3-home\" (UniqueName: \"kubernetes.io/empty-dir/cf4cf2fc-612a-4f5c-a596-543036dde7e3-home\") pod \"cf4cf2fc-612a-4f5c-a596-543036dde7e3\" (UID: \"cf4cf2fc-612a-4f5c-a596-543036dde7e3\") "
Nov 03 09:56:56.265133 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:56.265099    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf4cf2fc-612a-4f5c-a596-543036dde7e3-kube-api-access-6mnr9\" (UniqueName: \"kubernetes.io/projected/cf4cf2fc-612a-4f5c-a596-543036dde7e3-kube-api-access-6mnr9\") pod \"cf4cf2fc-612a-4f5c-a596-543036dde7e3\" (UID: \"cf4cf2fc-612a-4f5c-a596-543036dde7e3\") "
Nov 03 09:56:56.265133 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:56.265130    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf4cf2fc-612a-4f5c-a596-543036dde7e3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf4cf2fc-612a-4f5c-a596-543036dde7e3-dshm\") pod \"cf4cf2fc-612a-4f5c-a596-543036dde7e3\" (UID: \"cf4cf2fc-612a-4f5c-a596-543036dde7e3\") "
Nov 03 09:56:56.265395 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:56.265182    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf4cf2fc-612a-4f5c-a596-543036dde7e3-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf4cf2fc-612a-4f5c-a596-543036dde7e3-tls-certs\") pod \"cf4cf2fc-612a-4f5c-a596-543036dde7e3\" (UID: \"cf4cf2fc-612a-4f5c-a596-543036dde7e3\") "
Nov 03 09:56:56.265395 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:56.265248    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf4cf2fc-612a-4f5c-a596-543036dde7e3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf4cf2fc-612a-4f5c-a596-543036dde7e3-tmp-dir\") pod \"cf4cf2fc-612a-4f5c-a596-543036dde7e3\" (UID: \"cf4cf2fc-612a-4f5c-a596-543036dde7e3\") "
Nov 03 09:56:56.265395 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:56.265304    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf4cf2fc-612a-4f5c-a596-543036dde7e3-home" pod "cf4cf2fc-612a-4f5c-a596-543036dde7e3" (UID: "cf4cf2fc-612a-4f5c-a596-543036dde7e3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:56:56.265395 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:56.265346    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf4cf2fc-612a-4f5c-a596-543036dde7e3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf4cf2fc-612a-4f5c-a596-543036dde7e3-model-cache\") pod \"cf4cf2fc-612a-4f5c-a596-543036dde7e3\" (UID: \"cf4cf2fc-612a-4f5c-a596-543036dde7e3\") "
Nov 03 09:56:56.265607 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:56.265566    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf4cf2fc-612a-4f5c-a596-543036dde7e3-tmp-dir" pod "cf4cf2fc-612a-4f5c-a596-543036dde7e3" (UID: "cf4cf2fc-612a-4f5c-a596-543036dde7e3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:56:56.265663 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:56.265630    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf4cf2fc-612a-4f5c-a596-543036dde7e3-model-cache" pod "cf4cf2fc-612a-4f5c-a596-543036dde7e3" (UID: "cf4cf2fc-612a-4f5c-a596-543036dde7e3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:56:56.265740 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:56.265726    4473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf4cf2fc-612a-4f5c-a596-543036dde7e3-model-cache\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:56:56.265778 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:56.265744    4473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf4cf2fc-612a-4f5c-a596-543036dde7e3-home\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:56:56.265825 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:56.265776    4473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf4cf2fc-612a-4f5c-a596-543036dde7e3-tmp-dir\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:56:56.267571 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:56.267544    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf4cf2fc-612a-4f5c-a596-543036dde7e3-dshm" pod "cf4cf2fc-612a-4f5c-a596-543036dde7e3" (UID: "cf4cf2fc-612a-4f5c-a596-543036dde7e3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:56:56.267853 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:56.267833    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf4cf2fc-612a-4f5c-a596-543036dde7e3-kube-api-access-6mnr9" pod "cf4cf2fc-612a-4f5c-a596-543036dde7e3" (UID: "cf4cf2fc-612a-4f5c-a596-543036dde7e3"). InnerVolumeSpecName "kube-api-access-6mnr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:56:56.267853 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:56.267839    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf4cf2fc-612a-4f5c-a596-543036dde7e3-tls-certs" pod "cf4cf2fc-612a-4f5c-a596-543036dde7e3" (UID: "cf4cf2fc-612a-4f5c-a596-543036dde7e3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:56:56.366904 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:56.366871    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mnr9\" (UniqueName: \"kubernetes.io/projected/cf4cf2fc-612a-4f5c-a596-543036dde7e3-kube-api-access-6mnr9\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:56:56.366904 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:56.366901    4473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf4cf2fc-612a-4f5c-a596-543036dde7e3-dshm\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:56:56.366904 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:56.366911    4473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf4cf2fc-612a-4f5c-a596-543036dde7e3-tls-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:56:56.895808 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:56.895776    4473 generic.go:356] "Generic (PLEG): container finished" podID="cf4cf2fc-612a-4f5c-a596-543036dde7e3" containerID="1a185f8717c5d7f7d5f232ad5e2c83487feb26ce7edb70a70b31dcbe11f71cfd" exitCode=0
Nov 03 09:56:56.896016 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:56.895843    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd"
Nov 03 09:56:56.896016 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:56.895870    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd" event={"ID":"cf4cf2fc-612a-4f5c-a596-543036dde7e3","Type":"ContainerDied","Data":"1a185f8717c5d7f7d5f232ad5e2c83487feb26ce7edb70a70b31dcbe11f71cfd"}
Nov 03 09:56:56.896016 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:56.895915    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd" event={"ID":"cf4cf2fc-612a-4f5c-a596-543036dde7e3","Type":"ContainerDied","Data":"457f4e80e89340a0f74614bb47b2cec7be2aedc8af3a676ef77359e897644048"}
Nov 03 09:56:56.896016 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:56.895931    4473 scope.go:122] "RemoveContainer" containerID="1a185f8717c5d7f7d5f232ad5e2c83487feb26ce7edb70a70b31dcbe11f71cfd"
Nov 03 09:56:56.905622 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:56.905608    4473 scope.go:122] "RemoveContainer" containerID="1a185f8717c5d7f7d5f232ad5e2c83487feb26ce7edb70a70b31dcbe11f71cfd"
Nov 03 09:56:56.905899 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:56:56.905878    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a185f8717c5d7f7d5f232ad5e2c83487feb26ce7edb70a70b31dcbe11f71cfd\": container with ID starting with 1a185f8717c5d7f7d5f232ad5e2c83487feb26ce7edb70a70b31dcbe11f71cfd not found: ID does not exist" containerID="1a185f8717c5d7f7d5f232ad5e2c83487feb26ce7edb70a70b31dcbe11f71cfd"
Nov 03 09:56:56.905965 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:56.905909    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a185f8717c5d7f7d5f232ad5e2c83487feb26ce7edb70a70b31dcbe11f71cfd"} err="failed to get container status \"1a185f8717c5d7f7d5f232ad5e2c83487feb26ce7edb70a70b31dcbe11f71cfd\": rpc error: code = NotFound desc = could not find container \"1a185f8717c5d7f7d5f232ad5e2c83487feb26ce7edb70a70b31dcbe11f71cfd\": container with ID starting with 1a185f8717c5d7f7d5f232ad5e2c83487feb26ce7edb70a70b31dcbe11f71cfd not found: ID does not exist"
Nov 03 09:56:57.051211 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:57.051150    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc60feb6-2jr79"]
Nov 03 09:56:57.053900 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:57.053865    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc60feb6-2jr79"]
Nov 03 09:56:57.064230 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:57.064200    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd"]
Nov 03 09:56:57.069389 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:57.069361    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-77207e/simulator-w2-77207e-kserve-74458c56fc-bssxd"]
Nov 03 09:56:57.868882 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:57.868845    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3adfe3ef-a39a-4536-b2f1-c2c847341b66" path="/var/lib/kubelet/pods/3adfe3ef-a39a-4536-b2f1-c2c847341b66/volumes"
Nov 03 09:56:57.869283 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:57.869269    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf4cf2fc-612a-4f5c-a596-543036dde7e3" path="/var/lib/kubelet/pods/cf4cf2fc-612a-4f5c-a596-543036dde7e3/volumes"
Nov 03 09:56:59.213912 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:59.213881    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e3864c4-p9tv6"]
Nov 03 09:56:59.214370 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:59.214356    4473 cpu_manager.go:414] "containerMap: removing container" podUID="a7c91a81-8f4f-4161-9189-01d948f1b55c" containerName="revoke-keys"
Nov 03 09:56:59.214419 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:59.214371    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7c91a81-8f4f-4161-9189-01d948f1b55c" containerName="revoke-keys"
Nov 03 09:56:59.214419 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:59.214381    4473 cpu_manager.go:414] "containerMap: removing container" podUID="cf4cf2fc-612a-4f5c-a596-543036dde7e3" containerName="main"
Nov 03 09:56:59.214419 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:59.214386    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf4cf2fc-612a-4f5c-a596-543036dde7e3" containerName="main"
Nov 03 09:56:59.214514 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:59.214445    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7c91a81-8f4f-4161-9189-01d948f1b55c" containerName="revoke-keys"
Nov 03 09:56:59.214514 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:59.214453    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf4cf2fc-612a-4f5c-a596-543036dde7e3" containerName="main"
Nov 03 09:56:59.218977 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:59.218935    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e3864c4-p9tv6"
Nov 03 09:56:59.275391 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:59.275362    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e3864c4-p9tv6"]
Nov 03 09:56:59.302009 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:59.301967    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8582e32-8742-478d-9cf8-67a8488ab67c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e3864c4-p9tv6\" (UID: \"d8582e32-8742-478d-9cf8-67a8488ab67c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e3864c4-p9tv6"
Nov 03 09:56:59.302179 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:59.302091    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vm7j\" (UniqueName: \"kubernetes.io/projected/d8582e32-8742-478d-9cf8-67a8488ab67c-kube-api-access-4vm7j\") pod \"maas-api-revoke-sub-4e3864c4-p9tv6\" (UID: \"d8582e32-8742-478d-9cf8-67a8488ab67c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e3864c4-p9tv6"
Nov 03 09:56:59.302223 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:59.302197    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8582e32-8742-478d-9cf8-67a8488ab67c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e3864c4-p9tv6\" (UID: \"d8582e32-8742-478d-9cf8-67a8488ab67c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e3864c4-p9tv6"
Nov 03 09:56:59.322452 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:59.322425    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 09:56:59.322621 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:59.322513    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wj4kg\""
Nov 03 09:56:59.322621 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:59.322600    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 09:56:59.322621 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:59.322617    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 09:56:59.403222 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:59.403192    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8582e32-8742-478d-9cf8-67a8488ab67c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e3864c4-p9tv6\" (UID: \"d8582e32-8742-478d-9cf8-67a8488ab67c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e3864c4-p9tv6"
Nov 03 09:56:59.403392 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:59.403255    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vm7j\" (UniqueName: \"kubernetes.io/projected/d8582e32-8742-478d-9cf8-67a8488ab67c-kube-api-access-4vm7j\") pod \"maas-api-revoke-sub-4e3864c4-p9tv6\" (UID: \"d8582e32-8742-478d-9cf8-67a8488ab67c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e3864c4-p9tv6"
Nov 03 09:56:59.403392 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:59.403314    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8582e32-8742-478d-9cf8-67a8488ab67c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e3864c4-p9tv6\" (UID: \"d8582e32-8742-478d-9cf8-67a8488ab67c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e3864c4-p9tv6"
Nov 03 09:56:59.404095 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:59.404030    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8582e32-8742-478d-9cf8-67a8488ab67c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e3864c4-p9tv6\" (UID: \"d8582e32-8742-478d-9cf8-67a8488ab67c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e3864c4-p9tv6"
Nov 03 09:56:59.405797 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:59.405774    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8582e32-8742-478d-9cf8-67a8488ab67c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e3864c4-p9tv6\" (UID: \"d8582e32-8742-478d-9cf8-67a8488ab67c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e3864c4-p9tv6"
Nov 03 09:56:59.462166 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:59.462138    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vm7j\" (UniqueName: \"kubernetes.io/projected/d8582e32-8742-478d-9cf8-67a8488ab67c-kube-api-access-4vm7j\") pod \"maas-api-revoke-sub-4e3864c4-p9tv6\" (UID: \"d8582e32-8742-478d-9cf8-67a8488ab67c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e3864c4-p9tv6"
Nov 03 09:56:59.531861 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:59.531784    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e3864c4-p9tv6"
Nov 03 09:56:59.927585 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:56:59.927548    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e3864c4-p9tv6"]
Nov 03 09:56:59.928356 ip-10-0-139-73 kubenswrapper[4473]: W1103 09:56:59.928308    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8582e32_8742_478d_9cf8_67a8488ab67c.slice/crio-c659cc7651a99680d3495a794416e63756e43c3103f05be1d96ba29fddad1d94 WatchSource:0}: Error finding container c659cc7651a99680d3495a794416e63756e43c3103f05be1d96ba29fddad1d94: Status 404 returned error can't find the container with id c659cc7651a99680d3495a794416e63756e43c3103f05be1d96ba29fddad1d94
Nov 03 09:57:00.733043 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:00.733015    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-910d50-7db8cfbf6c-w8z4h"]
Nov 03 09:57:00.937543 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:00.937451    4473 generic.go:356] "Generic (PLEG): container finished" podID="d8582e32-8742-478d-9cf8-67a8488ab67c" containerID="4a6d51bdbfaa8778e6ee00d216c5ef0056bc9e4653481a6a5bb8dd36676e0110" exitCode=0
Nov 03 09:57:00.937839 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:00.937654    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e3864c4-p9tv6" event={"ID":"d8582e32-8742-478d-9cf8-67a8488ab67c","Type":"ContainerDied","Data":"4a6d51bdbfaa8778e6ee00d216c5ef0056bc9e4653481a6a5bb8dd36676e0110"}
Nov 03 09:57:00.937839 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:00.937708    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e3864c4-p9tv6" event={"ID":"d8582e32-8742-478d-9cf8-67a8488ab67c","Type":"ContainerStarted","Data":"c659cc7651a99680d3495a794416e63756e43c3103f05be1d96ba29fddad1d94"}
Nov 03 09:57:02.130346 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:02.130325    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e3864c4-p9tv6"
Nov 03 09:57:02.249628 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:02.249555    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8582e32-8742-478d-9cf8-67a8488ab67c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8582e32-8742-478d-9cf8-67a8488ab67c-maas-api-openshift-ca\") pod \"d8582e32-8742-478d-9cf8-67a8488ab67c\" (UID: \"d8582e32-8742-478d-9cf8-67a8488ab67c\") "
Nov 03 09:57:02.249794 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:02.249666    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8582e32-8742-478d-9cf8-67a8488ab67c-kube-api-access-4vm7j\" (UniqueName: \"kubernetes.io/projected/d8582e32-8742-478d-9cf8-67a8488ab67c-kube-api-access-4vm7j\") pod \"d8582e32-8742-478d-9cf8-67a8488ab67c\" (UID: \"d8582e32-8742-478d-9cf8-67a8488ab67c\") "
Nov 03 09:57:02.249794 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:02.249737    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8582e32-8742-478d-9cf8-67a8488ab67c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8582e32-8742-478d-9cf8-67a8488ab67c-maas-api-xks-ca\") pod \"d8582e32-8742-478d-9cf8-67a8488ab67c\" (UID: \"d8582e32-8742-478d-9cf8-67a8488ab67c\") "
Nov 03 09:57:02.250012 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:02.249986    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8582e32-8742-478d-9cf8-67a8488ab67c-maas-api-openshift-ca" pod "d8582e32-8742-478d-9cf8-67a8488ab67c" (UID: "d8582e32-8742-478d-9cf8-67a8488ab67c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:57:02.250101 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:02.250080    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8582e32-8742-478d-9cf8-67a8488ab67c-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:02.252712 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:02.252658    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8582e32-8742-478d-9cf8-67a8488ab67c-maas-api-xks-ca" pod "d8582e32-8742-478d-9cf8-67a8488ab67c" (UID: "d8582e32-8742-478d-9cf8-67a8488ab67c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:57:02.252712 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:02.252668    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8582e32-8742-478d-9cf8-67a8488ab67c-kube-api-access-4vm7j" pod "d8582e32-8742-478d-9cf8-67a8488ab67c" (UID: "d8582e32-8742-478d-9cf8-67a8488ab67c"). InnerVolumeSpecName "kube-api-access-4vm7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:57:02.350729 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:02.350698    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8582e32-8742-478d-9cf8-67a8488ab67c-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:02.350729 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:02.350726    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vm7j\" (UniqueName: \"kubernetes.io/projected/d8582e32-8742-478d-9cf8-67a8488ab67c-kube-api-access-4vm7j\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:02.952096 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:02.952068    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e3864c4-p9tv6"
Nov 03 09:57:02.952268 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:02.952067    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e3864c4-p9tv6" event={"ID":"d8582e32-8742-478d-9cf8-67a8488ab67c","Type":"ContainerDied","Data":"c659cc7651a99680d3495a794416e63756e43c3103f05be1d96ba29fddad1d94"}
Nov 03 09:57:02.952268 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:02.952196    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c659cc7651a99680d3495a794416e63756e43c3103f05be1d96ba29fddad1d94"
Nov 03 09:57:05.734114 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:05.734052    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-910d50-7db8cfbf6c-w8z4h" podUID="960d4e1b-7409-4a75-83f4-26289cb2b694" containerName="payload-pre-processing" containerID="cri-o://14e14f7a4445a22d1a082137713ea9d092a41b39400dbf88f167bdfd57c632a1" gracePeriod=25
Nov 03 09:57:05.742765 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:05.742733    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj" podUID="6afd6431-8c5e-4a46-b505-270efaaacd1c" containerName="main" containerID="cri-o://a03b4f719ec45cdfa5aa271222818dff5f7438cbc4d891845f0cd8e7394b2c21" gracePeriod=45
Nov 03 09:57:05.791198 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:05.791163    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn" podUID="035b0461-2dfa-42f4-8ea7-2d1911cddca9" containerName="main" containerID="cri-o://66b8d0a5f7fd81910c4dc73e8c6cc0836b43a2cb8c41b98c1207c16860005d9d" gracePeriod=45
Nov 03 09:57:05.903143 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:05.903109    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn" podUID="035b0461-2dfa-42f4-8ea7-2d1911cddca9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": dial tcp 10.133.0.71:8000: connect: connection refused"
Nov 03 09:57:05.974406 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:05.974012    4473 generic.go:356] "Generic (PLEG): container finished" podID="035b0461-2dfa-42f4-8ea7-2d1911cddca9" containerID="66b8d0a5f7fd81910c4dc73e8c6cc0836b43a2cb8c41b98c1207c16860005d9d" exitCode=0
Nov 03 09:57:05.974406 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:05.974161    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn" event={"ID":"035b0461-2dfa-42f4-8ea7-2d1911cddca9","Type":"ContainerDied","Data":"66b8d0a5f7fd81910c4dc73e8c6cc0836b43a2cb8c41b98c1207c16860005d9d"}
Nov 03 09:57:05.975735 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:05.975643    4473 generic.go:356] "Generic (PLEG): container finished" podID="6afd6431-8c5e-4a46-b505-270efaaacd1c" containerID="a03b4f719ec45cdfa5aa271222818dff5f7438cbc4d891845f0cd8e7394b2c21" exitCode=0
Nov 03 09:57:05.975735 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:05.975700    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj" event={"ID":"6afd6431-8c5e-4a46-b505-270efaaacd1c","Type":"ContainerDied","Data":"a03b4f719ec45cdfa5aa271222818dff5f7438cbc4d891845f0cd8e7394b2c21"}
Nov 03 09:57:06.041646 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.041621    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj"
Nov 03 09:57:06.079348 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.079323    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn"
Nov 03 09:57:06.190314 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.190285    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6afd6431-8c5e-4a46-b505-270efaaacd1c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6afd6431-8c5e-4a46-b505-270efaaacd1c-dshm\") pod \"6afd6431-8c5e-4a46-b505-270efaaacd1c\" (UID: \"6afd6431-8c5e-4a46-b505-270efaaacd1c\") "
Nov 03 09:57:06.190491 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.190318    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/035b0461-2dfa-42f4-8ea7-2d1911cddca9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/035b0461-2dfa-42f4-8ea7-2d1911cddca9-model-cache\") pod \"035b0461-2dfa-42f4-8ea7-2d1911cddca9\" (UID: \"035b0461-2dfa-42f4-8ea7-2d1911cddca9\") "
Nov 03 09:57:06.190491 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.190359    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6afd6431-8c5e-4a46-b505-270efaaacd1c-kube-api-access-544mx\" (UniqueName: \"kubernetes.io/projected/6afd6431-8c5e-4a46-b505-270efaaacd1c-kube-api-access-544mx\") pod \"6afd6431-8c5e-4a46-b505-270efaaacd1c\" (UID: \"6afd6431-8c5e-4a46-b505-270efaaacd1c\") "
Nov 03 09:57:06.190491 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.190377    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/035b0461-2dfa-42f4-8ea7-2d1911cddca9-home\" (UniqueName: \"kubernetes.io/empty-dir/035b0461-2dfa-42f4-8ea7-2d1911cddca9-home\") pod \"035b0461-2dfa-42f4-8ea7-2d1911cddca9\" (UID: \"035b0461-2dfa-42f4-8ea7-2d1911cddca9\") "
Nov 03 09:57:06.190491 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.190413    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/035b0461-2dfa-42f4-8ea7-2d1911cddca9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/035b0461-2dfa-42f4-8ea7-2d1911cddca9-dshm\") pod \"035b0461-2dfa-42f4-8ea7-2d1911cddca9\" (UID: \"035b0461-2dfa-42f4-8ea7-2d1911cddca9\") "
Nov 03 09:57:06.190491 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.190459    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/035b0461-2dfa-42f4-8ea7-2d1911cddca9-tls-certs\" (UniqueName: \"kubernetes.io/secret/035b0461-2dfa-42f4-8ea7-2d1911cddca9-tls-certs\") pod \"035b0461-2dfa-42f4-8ea7-2d1911cddca9\" (UID: \"035b0461-2dfa-42f4-8ea7-2d1911cddca9\") "
Nov 03 09:57:06.191262 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.190502    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6afd6431-8c5e-4a46-b505-270efaaacd1c-home\" (UniqueName: \"kubernetes.io/empty-dir/6afd6431-8c5e-4a46-b505-270efaaacd1c-home\") pod \"6afd6431-8c5e-4a46-b505-270efaaacd1c\" (UID: \"6afd6431-8c5e-4a46-b505-270efaaacd1c\") "
Nov 03 09:57:06.191262 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.190546    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6afd6431-8c5e-4a46-b505-270efaaacd1c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6afd6431-8c5e-4a46-b505-270efaaacd1c-model-cache\") pod \"6afd6431-8c5e-4a46-b505-270efaaacd1c\" (UID: \"6afd6431-8c5e-4a46-b505-270efaaacd1c\") "
Nov 03 09:57:06.191262 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.190582    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/035b0461-2dfa-42f4-8ea7-2d1911cddca9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/035b0461-2dfa-42f4-8ea7-2d1911cddca9-tmp-dir\") pod \"035b0461-2dfa-42f4-8ea7-2d1911cddca9\" (UID: \"035b0461-2dfa-42f4-8ea7-2d1911cddca9\") "
Nov 03 09:57:06.191262 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.190689    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6afd6431-8c5e-4a46-b505-270efaaacd1c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6afd6431-8c5e-4a46-b505-270efaaacd1c-tmp-dir\") pod \"6afd6431-8c5e-4a46-b505-270efaaacd1c\" (UID: \"6afd6431-8c5e-4a46-b505-270efaaacd1c\") "
Nov 03 09:57:06.191262 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.190753    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6afd6431-8c5e-4a46-b505-270efaaacd1c-tls-certs\" (UniqueName: \"kubernetes.io/secret/6afd6431-8c5e-4a46-b505-270efaaacd1c-tls-certs\") pod \"6afd6431-8c5e-4a46-b505-270efaaacd1c\" (UID: \"6afd6431-8c5e-4a46-b505-270efaaacd1c\") "
Nov 03 09:57:06.191262 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.190758    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/035b0461-2dfa-42f4-8ea7-2d1911cddca9-home" pod "035b0461-2dfa-42f4-8ea7-2d1911cddca9" (UID: "035b0461-2dfa-42f4-8ea7-2d1911cddca9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:57:06.191262 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.190831    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/035b0461-2dfa-42f4-8ea7-2d1911cddca9-kube-api-access-8mcnc\" (UniqueName: \"kubernetes.io/projected/035b0461-2dfa-42f4-8ea7-2d1911cddca9-kube-api-access-8mcnc\") pod \"035b0461-2dfa-42f4-8ea7-2d1911cddca9\" (UID: \"035b0461-2dfa-42f4-8ea7-2d1911cddca9\") "
Nov 03 09:57:06.191262 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.191018    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6afd6431-8c5e-4a46-b505-270efaaacd1c-home" pod "6afd6431-8c5e-4a46-b505-270efaaacd1c" (UID: "6afd6431-8c5e-4a46-b505-270efaaacd1c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:57:06.191915 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.191444    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/035b0461-2dfa-42f4-8ea7-2d1911cddca9-model-cache" pod "035b0461-2dfa-42f4-8ea7-2d1911cddca9" (UID: "035b0461-2dfa-42f4-8ea7-2d1911cddca9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:57:06.191915 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.191594    4473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6afd6431-8c5e-4a46-b505-270efaaacd1c-home\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:06.191915 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.191617    4473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/035b0461-2dfa-42f4-8ea7-2d1911cddca9-home\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:06.191915 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.191808    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/035b0461-2dfa-42f4-8ea7-2d1911cddca9-tmp-dir" pod "035b0461-2dfa-42f4-8ea7-2d1911cddca9" (UID: "035b0461-2dfa-42f4-8ea7-2d1911cddca9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:57:06.192193 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.192019    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6afd6431-8c5e-4a46-b505-270efaaacd1c-model-cache" pod "6afd6431-8c5e-4a46-b505-270efaaacd1c" (UID: "6afd6431-8c5e-4a46-b505-270efaaacd1c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:57:06.192193 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.192091    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6afd6431-8c5e-4a46-b505-270efaaacd1c-tmp-dir" pod "6afd6431-8c5e-4a46-b505-270efaaacd1c" (UID: "6afd6431-8c5e-4a46-b505-270efaaacd1c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:57:06.193443 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.193416    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6afd6431-8c5e-4a46-b505-270efaaacd1c-kube-api-access-544mx" pod "6afd6431-8c5e-4a46-b505-270efaaacd1c" (UID: "6afd6431-8c5e-4a46-b505-270efaaacd1c"). InnerVolumeSpecName "kube-api-access-544mx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:57:06.193867 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.193831    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/035b0461-2dfa-42f4-8ea7-2d1911cddca9-kube-api-access-8mcnc" pod "035b0461-2dfa-42f4-8ea7-2d1911cddca9" (UID: "035b0461-2dfa-42f4-8ea7-2d1911cddca9"). InnerVolumeSpecName "kube-api-access-8mcnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:57:06.194177 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.194152    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6afd6431-8c5e-4a46-b505-270efaaacd1c-dshm" pod "6afd6431-8c5e-4a46-b505-270efaaacd1c" (UID: "6afd6431-8c5e-4a46-b505-270efaaacd1c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:57:06.194417 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.194393    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/035b0461-2dfa-42f4-8ea7-2d1911cddca9-tls-certs" pod "035b0461-2dfa-42f4-8ea7-2d1911cddca9" (UID: "035b0461-2dfa-42f4-8ea7-2d1911cddca9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:57:06.194588 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.194567    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/035b0461-2dfa-42f4-8ea7-2d1911cddca9-dshm" pod "035b0461-2dfa-42f4-8ea7-2d1911cddca9" (UID: "035b0461-2dfa-42f4-8ea7-2d1911cddca9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:57:06.194650 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.194599    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6afd6431-8c5e-4a46-b505-270efaaacd1c-tls-certs" pod "6afd6431-8c5e-4a46-b505-270efaaacd1c" (UID: "6afd6431-8c5e-4a46-b505-270efaaacd1c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:57:06.292835 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.292756    4473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6afd6431-8c5e-4a46-b505-270efaaacd1c-model-cache\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:06.292835 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.292810    4473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/035b0461-2dfa-42f4-8ea7-2d1911cddca9-tmp-dir\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:06.292835 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.292826    4473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6afd6431-8c5e-4a46-b505-270efaaacd1c-tmp-dir\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:06.293179 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.292838    4473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6afd6431-8c5e-4a46-b505-270efaaacd1c-tls-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:06.293179 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.292852    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mcnc\" (UniqueName: \"kubernetes.io/projected/035b0461-2dfa-42f4-8ea7-2d1911cddca9-kube-api-access-8mcnc\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:06.293179 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.292865    4473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6afd6431-8c5e-4a46-b505-270efaaacd1c-dshm\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:06.293179 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.292876    4473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/035b0461-2dfa-42f4-8ea7-2d1911cddca9-model-cache\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:06.293179 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.292887    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-544mx\" (UniqueName: \"kubernetes.io/projected/6afd6431-8c5e-4a46-b505-270efaaacd1c-kube-api-access-544mx\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:06.293179 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.292900    4473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/035b0461-2dfa-42f4-8ea7-2d1911cddca9-dshm\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:06.293179 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.292913    4473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/035b0461-2dfa-42f4-8ea7-2d1911cddca9-tls-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:06.772509 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.772469    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"]
Nov 03 09:57:06.773002 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.772709    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5" podUID="a697a58e-ba2e-4bff-a9fb-0d749cc6e044" containerName="istio-proxy" containerID="cri-o://921e998ef623e19db1396991c181984dfe4df354c68b7384184f5eff2c857ebc" gracePeriod=30
Nov 03 09:57:06.983592 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.983556    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn"
Nov 03 09:57:06.983592 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.983579    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn" event={"ID":"035b0461-2dfa-42f4-8ea7-2d1911cddca9","Type":"ContainerDied","Data":"b53cbffc02d3c5c1d8d6160475d10f8430c8986fdd652c58a1c41f4878121d43"}
Nov 03 09:57:06.983841 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.983625    4473 scope.go:122] "RemoveContainer" containerID="66b8d0a5f7fd81910c4dc73e8c6cc0836b43a2cb8c41b98c1207c16860005d9d"
Nov 03 09:57:06.985131 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.985102    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj"
Nov 03 09:57:06.985262 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.985099    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj" event={"ID":"6afd6431-8c5e-4a46-b505-270efaaacd1c","Type":"ContainerDied","Data":"825a4dba0e4a26e963fe5518bbac2d82628ed062b78136adb8c1cad324c0f5f8"}
Nov 03 09:57:06.994742 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:06.994721    4473 scope.go:122] "RemoveContainer" containerID="a03b4f719ec45cdfa5aa271222818dff5f7438cbc4d891845f0cd8e7394b2c21"
Nov 03 09:57:07.152898 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:07.152868    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn"]
Nov 03 09:57:07.155300 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:07.155281    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-910d50/distinct-w0-910d50-kserve-679d4f866-jxdtn"]
Nov 03 09:57:07.266557 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:07.266522    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj"]
Nov 03 09:57:07.276276 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:07.276252    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-910d50/distinct-2-w0-910d50-kserve-56d8cd54cc-6bllj"]
Nov 03 09:57:07.868642 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:07.868613    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="035b0461-2dfa-42f4-8ea7-2d1911cddca9" path="/var/lib/kubelet/pods/035b0461-2dfa-42f4-8ea7-2d1911cddca9/volumes"
Nov 03 09:57:07.869190 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:07.869174    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6afd6431-8c5e-4a46-b505-270efaaacd1c" path="/var/lib/kubelet/pods/6afd6431-8c5e-4a46-b505-270efaaacd1c/volumes"
Nov 03 09:57:08.053081 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:08.053055    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18f4b584-t4dqg"]
Nov 03 09:57:08.103333 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:08.103297    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18f4b584-t4dqg"]
Nov 03 09:57:09.527858 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:09.527816    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-910d50-7db8cfbf6c-w8z4h" podUID="960d4e1b-7409-4a75-83f4-26289cb2b694" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:57:09.868184 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:09.868154    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2" path="/var/lib/kubelet/pods/8d99a47e-d30d-4f87-a0a7-d3e3de0d5bd2/volumes"
Nov 03 09:57:11.742532 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:11.742492    4473 patch_prober.go:29] interesting pod/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 09:57:11.743020 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:11.742563    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5" podUID="a697a58e-ba2e-4bff-a9fb-0d749cc6e044" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 09:57:12.015250 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:12.015217    4473 generic.go:356] "Generic (PLEG): container finished" podID="a697a58e-ba2e-4bff-a9fb-0d749cc6e044" containerID="921e998ef623e19db1396991c181984dfe4df354c68b7384184f5eff2c857ebc" exitCode=0
Nov 03 09:57:12.015406 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:12.015324    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5" event={"ID":"a697a58e-ba2e-4bff-a9fb-0d749cc6e044","Type":"ContainerDied","Data":"921e998ef623e19db1396991c181984dfe4df354c68b7384184f5eff2c857ebc"}
Nov 03 09:57:12.029282 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:12.029258    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"
Nov 03 09:57:12.155056 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:12.155030    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-istio-podinfo\") pod \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\" (UID: \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\") "
Nov 03 09:57:12.155249 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:12.155073    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-workload-socket\") pod \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\" (UID: \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\") "
Nov 03 09:57:12.155249 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:12.155135    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-istio-token\" (UniqueName: \"kubernetes.io/projected/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-istio-token\") pod \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\" (UID: \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\") "
Nov 03 09:57:12.155249 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:12.155165    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-istio-envoy\") pod \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\" (UID: \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\") "
Nov 03 09:57:12.155249 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:12.155205    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-istio-data\") pod \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\" (UID: \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\") "
Nov 03 09:57:12.155249 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:12.155234    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-istiod-ca-cert\") pod \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\" (UID: \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\") "
Nov 03 09:57:12.155539 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:12.155269    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-workload-certs\") pod \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\" (UID: \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\") "
Nov 03 09:57:12.155539 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:12.155406    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-kube-api-access-xzh4c\" (UniqueName: \"kubernetes.io/projected/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-kube-api-access-xzh4c\") pod \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\" (UID: \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\") "
Nov 03 09:57:12.155539 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:12.155410    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-workload-socket" pod "a697a58e-ba2e-4bff-a9fb-0d749cc6e044" (UID: "a697a58e-ba2e-4bff-a9fb-0d749cc6e044"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:57:12.155539 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:12.155461    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-credential-socket\") pod \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\" (UID: \"a697a58e-ba2e-4bff-a9fb-0d749cc6e044\") "
Nov 03 09:57:12.155539 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:12.155514    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-istio-data" pod "a697a58e-ba2e-4bff-a9fb-0d749cc6e044" (UID: "a697a58e-ba2e-4bff-a9fb-0d749cc6e044"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:57:12.155803 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:12.155634    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-workload-certs" pod "a697a58e-ba2e-4bff-a9fb-0d749cc6e044" (UID: "a697a58e-ba2e-4bff-a9fb-0d749cc6e044"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:57:12.155803 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:12.155654    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-istiod-ca-cert" pod "a697a58e-ba2e-4bff-a9fb-0d749cc6e044" (UID: "a697a58e-ba2e-4bff-a9fb-0d749cc6e044"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:57:12.155929 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:12.155861    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-credential-socket" pod "a697a58e-ba2e-4bff-a9fb-0d749cc6e044" (UID: "a697a58e-ba2e-4bff-a9fb-0d749cc6e044"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:57:12.155929 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:12.155865    4473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-istio-data\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:12.155929 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:12.155902    4473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-istiod-ca-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:12.155929 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:12.155919    4473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-workload-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:12.156161 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:12.155937    4473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-workload-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:12.157516 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:12.157489    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-istio-envoy" pod "a697a58e-ba2e-4bff-a9fb-0d749cc6e044" (UID: "a697a58e-ba2e-4bff-a9fb-0d749cc6e044"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:57:12.157739 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:12.157719    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-kube-api-access-xzh4c" pod "a697a58e-ba2e-4bff-a9fb-0d749cc6e044" (UID: "a697a58e-ba2e-4bff-a9fb-0d749cc6e044"). InnerVolumeSpecName "kube-api-access-xzh4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:57:12.158006 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:12.157985    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-istio-podinfo" pod "a697a58e-ba2e-4bff-a9fb-0d749cc6e044" (UID: "a697a58e-ba2e-4bff-a9fb-0d749cc6e044"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 09:57:12.158006 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:12.157986    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-istio-token" pod "a697a58e-ba2e-4bff-a9fb-0d749cc6e044" (UID: "a697a58e-ba2e-4bff-a9fb-0d749cc6e044"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:57:12.255054 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:12.255017    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx"]
Nov 03 09:57:12.257581 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:12.257549    4473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-istio-token\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:12.257581 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:12.257582    4473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-istio-envoy\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:12.257818 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:12.257598    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzh4c\" (UniqueName: \"kubernetes.io/projected/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-kube-api-access-xzh4c\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:12.257818 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:12.257615    4473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-credential-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:12.257818 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:12.257630    4473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a697a58e-ba2e-4bff-a9fb-0d749cc6e044-istio-podinfo\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:12.258181 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:12.258157    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6"]
Nov 03 09:57:13.022447 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:13.022416    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"
Nov 03 09:57:13.022447 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:13.022433    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5" event={"ID":"a697a58e-ba2e-4bff-a9fb-0d749cc6e044","Type":"ContainerDied","Data":"f2d0ae0697693af44d7af7babd5730cd9235ba078cd42f407839a81e94b37ad3"}
Nov 03 09:57:13.022990 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:13.022489    4473 scope.go:122] "RemoveContainer" containerID="921e998ef623e19db1396991c181984dfe4df354c68b7384184f5eff2c857ebc"
Nov 03 09:57:13.228063 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:13.228034    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"]
Nov 03 09:57:13.278295 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:13.278209    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-b196c8-openshift-default-58f7cccd45-wdnf5"]
Nov 03 09:57:13.868976 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:13.868915    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a697a58e-ba2e-4bff-a9fb-0d749cc6e044" path="/var/lib/kubelet/pods/a697a58e-ba2e-4bff-a9fb-0d749cc6e044/volumes"
Nov 03 09:57:16.938822 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:57:16.938789    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:57:19.528000 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:19.527963    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-910d50-7db8cfbf6c-w8z4h" podUID="960d4e1b-7409-4a75-83f4-26289cb2b694" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:57:20.975364 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:20.975333    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"]
Nov 03 09:57:20.975808 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:20.975529    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr" podUID="cc159b5a-6ae2-4f09-a74d-1ff15927d692" containerName="istio-proxy" containerID="cri-o://cd9aeb3769f2c937b77f31625d243052b7d238938466e2a983af477c93f9141b" gracePeriod=30
Nov 03 09:57:22.059701 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:22.059131    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-98d5dd-d6c54bc97-xhlw4"]
Nov 03 09:57:22.168433 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:22.168390    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-98d5dd-77466f785f-drltw"]
Nov 03 09:57:25.987336 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:25.987312    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-910d50-7db8cfbf6c-w8z4h"
Nov 03 09:57:25.990727 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:25.990661    4473 patch_prober.go:29] interesting pod/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": read tcp 10.133.0.2:55086->10.133.0.76:15021: read: connection reset by peer" start-of-body=
Nov 03 09:57:25.990858 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:25.990728    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr" podUID="cc159b5a-6ae2-4f09-a74d-1ff15927d692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": read tcp 10.133.0.2:55086->10.133.0.76:15021: read: connection reset by peer"
Nov 03 09:57:26.099423 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.099399    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/960d4e1b-7409-4a75-83f4-26289cb2b694-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/960d4e1b-7409-4a75-83f4-26289cb2b694-plugins-config-volume\") pod \"960d4e1b-7409-4a75-83f4-26289cb2b694\" (UID: \"960d4e1b-7409-4a75-83f4-26289cb2b694\") "
Nov 03 09:57:26.099567 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.099548    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/960d4e1b-7409-4a75-83f4-26289cb2b694-kube-api-access-hv7zl\" (UniqueName: \"kubernetes.io/projected/960d4e1b-7409-4a75-83f4-26289cb2b694-kube-api-access-hv7zl\") pod \"960d4e1b-7409-4a75-83f4-26289cb2b694\" (UID: \"960d4e1b-7409-4a75-83f4-26289cb2b694\") "
Nov 03 09:57:26.099760 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.099738    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/960d4e1b-7409-4a75-83f4-26289cb2b694-plugins-config-volume" pod "960d4e1b-7409-4a75-83f4-26289cb2b694" (UID: "960d4e1b-7409-4a75-83f4-26289cb2b694"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:57:26.100373 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.100347    4473 generic.go:356] "Generic (PLEG): container finished" podID="960d4e1b-7409-4a75-83f4-26289cb2b694" containerID="14e14f7a4445a22d1a082137713ea9d092a41b39400dbf88f167bdfd57c632a1" exitCode=0
Nov 03 09:57:26.100491 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.100399    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-910d50-7db8cfbf6c-w8z4h"
Nov 03 09:57:26.100491 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.100433    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-910d50-7db8cfbf6c-w8z4h" event={"ID":"960d4e1b-7409-4a75-83f4-26289cb2b694","Type":"ContainerDied","Data":"14e14f7a4445a22d1a082137713ea9d092a41b39400dbf88f167bdfd57c632a1"}
Nov 03 09:57:26.100609 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.100504    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-910d50-7db8cfbf6c-w8z4h" event={"ID":"960d4e1b-7409-4a75-83f4-26289cb2b694","Type":"ContainerDied","Data":"77e37edf41981fdaba3f753cbae7beef9a2e3fd4c727f062b0c9c69307928b8f"}
Nov 03 09:57:26.100609 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.100537    4473 scope.go:122] "RemoveContainer" containerID="14e14f7a4445a22d1a082137713ea9d092a41b39400dbf88f167bdfd57c632a1"
Nov 03 09:57:26.102220 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.102194    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/960d4e1b-7409-4a75-83f4-26289cb2b694-kube-api-access-hv7zl" pod "960d4e1b-7409-4a75-83f4-26289cb2b694" (UID: "960d4e1b-7409-4a75-83f4-26289cb2b694"). InnerVolumeSpecName "kube-api-access-hv7zl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:57:26.102722 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.102691    4473 generic.go:356] "Generic (PLEG): container finished" podID="cc159b5a-6ae2-4f09-a74d-1ff15927d692" containerID="cd9aeb3769f2c937b77f31625d243052b7d238938466e2a983af477c93f9141b" exitCode=0
Nov 03 09:57:26.102834 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.102727    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr" event={"ID":"cc159b5a-6ae2-4f09-a74d-1ff15927d692","Type":"ContainerDied","Data":"cd9aeb3769f2c937b77f31625d243052b7d238938466e2a983af477c93f9141b"}
Nov 03 09:57:26.152092 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.152064    4473 scope.go:122] "RemoveContainer" containerID="14e14f7a4445a22d1a082137713ea9d092a41b39400dbf88f167bdfd57c632a1"
Nov 03 09:57:26.152493 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:57:26.152471    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14e14f7a4445a22d1a082137713ea9d092a41b39400dbf88f167bdfd57c632a1\": container with ID starting with 14e14f7a4445a22d1a082137713ea9d092a41b39400dbf88f167bdfd57c632a1 not found: ID does not exist" containerID="14e14f7a4445a22d1a082137713ea9d092a41b39400dbf88f167bdfd57c632a1"
Nov 03 09:57:26.152591 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.152517    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14e14f7a4445a22d1a082137713ea9d092a41b39400dbf88f167bdfd57c632a1"} err="failed to get container status \"14e14f7a4445a22d1a082137713ea9d092a41b39400dbf88f167bdfd57c632a1\": rpc error: code = NotFound desc = could not find container \"14e14f7a4445a22d1a082137713ea9d092a41b39400dbf88f167bdfd57c632a1\": container with ID starting with 14e14f7a4445a22d1a082137713ea9d092a41b39400dbf88f167bdfd57c632a1 not found: ID does not exist"
Nov 03 09:57:26.200875 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.200844    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv7zl\" (UniqueName: \"kubernetes.io/projected/960d4e1b-7409-4a75-83f4-26289cb2b694-kube-api-access-hv7zl\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:26.200875 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.200875    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/960d4e1b-7409-4a75-83f4-26289cb2b694-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:26.236930 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.236907    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"
Nov 03 09:57:26.301851 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.301785    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc159b5a-6ae2-4f09-a74d-1ff15927d692-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc159b5a-6ae2-4f09-a74d-1ff15927d692-istiod-ca-cert\") pod \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\" (UID: \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\") "
Nov 03 09:57:26.302001 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.301864    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc159b5a-6ae2-4f09-a74d-1ff15927d692-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc159b5a-6ae2-4f09-a74d-1ff15927d692-workload-socket\") pod \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\" (UID: \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\") "
Nov 03 09:57:26.302001 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.301907    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc159b5a-6ae2-4f09-a74d-1ff15927d692-istio-token\" (UniqueName: \"kubernetes.io/projected/cc159b5a-6ae2-4f09-a74d-1ff15927d692-istio-token\") pod \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\" (UID: \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\") "
Nov 03 09:57:26.302001 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.301972    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc159b5a-6ae2-4f09-a74d-1ff15927d692-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc159b5a-6ae2-4f09-a74d-1ff15927d692-istio-podinfo\") pod \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\" (UID: \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\") "
Nov 03 09:57:26.302166 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.302016    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc159b5a-6ae2-4f09-a74d-1ff15927d692-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc159b5a-6ae2-4f09-a74d-1ff15927d692-workload-certs\") pod \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\" (UID: \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\") "
Nov 03 09:57:26.302166 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.302066    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc159b5a-6ae2-4f09-a74d-1ff15927d692-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc159b5a-6ae2-4f09-a74d-1ff15927d692-istio-data\") pod \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\" (UID: \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\") "
Nov 03 09:57:26.302166 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.302098    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc159b5a-6ae2-4f09-a74d-1ff15927d692-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc159b5a-6ae2-4f09-a74d-1ff15927d692-credential-socket\") pod \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\" (UID: \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\") "
Nov 03 09:57:26.302166 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.302132    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc159b5a-6ae2-4f09-a74d-1ff15927d692-workload-socket" pod "cc159b5a-6ae2-4f09-a74d-1ff15927d692" (UID: "cc159b5a-6ae2-4f09-a74d-1ff15927d692"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:57:26.302403 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.302174    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc159b5a-6ae2-4f09-a74d-1ff15927d692-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc159b5a-6ae2-4f09-a74d-1ff15927d692-istio-envoy\") pod \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\" (UID: \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\") "
Nov 03 09:57:26.302403 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.302208    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc159b5a-6ae2-4f09-a74d-1ff15927d692-kube-api-access-hqc9j\" (UniqueName: \"kubernetes.io/projected/cc159b5a-6ae2-4f09-a74d-1ff15927d692-kube-api-access-hqc9j\") pod \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\" (UID: \"cc159b5a-6ae2-4f09-a74d-1ff15927d692\") "
Nov 03 09:57:26.302403 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.302293    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc159b5a-6ae2-4f09-a74d-1ff15927d692-workload-certs" pod "cc159b5a-6ae2-4f09-a74d-1ff15927d692" (UID: "cc159b5a-6ae2-4f09-a74d-1ff15927d692"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:57:26.302403 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.302300    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc159b5a-6ae2-4f09-a74d-1ff15927d692-istiod-ca-cert" pod "cc159b5a-6ae2-4f09-a74d-1ff15927d692" (UID: "cc159b5a-6ae2-4f09-a74d-1ff15927d692"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:57:26.302638 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.302431    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc159b5a-6ae2-4f09-a74d-1ff15927d692-credential-socket" pod "cc159b5a-6ae2-4f09-a74d-1ff15927d692" (UID: "cc159b5a-6ae2-4f09-a74d-1ff15927d692"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:57:26.302638 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.302507    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc159b5a-6ae2-4f09-a74d-1ff15927d692-istio-data" pod "cc159b5a-6ae2-4f09-a74d-1ff15927d692" (UID: "cc159b5a-6ae2-4f09-a74d-1ff15927d692"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:57:26.302638 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.302586    4473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc159b5a-6ae2-4f09-a74d-1ff15927d692-istiod-ca-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:26.302638 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.302598    4473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc159b5a-6ae2-4f09-a74d-1ff15927d692-workload-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:26.302638 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.302608    4473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc159b5a-6ae2-4f09-a74d-1ff15927d692-workload-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:26.302638 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.302617    4473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc159b5a-6ae2-4f09-a74d-1ff15927d692-istio-data\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:26.302638 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.302626    4473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc159b5a-6ae2-4f09-a74d-1ff15927d692-credential-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:26.304322 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.304297    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc159b5a-6ae2-4f09-a74d-1ff15927d692-istio-token" pod "cc159b5a-6ae2-4f09-a74d-1ff15927d692" (UID: "cc159b5a-6ae2-4f09-a74d-1ff15927d692"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:57:26.304415 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.304326    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc159b5a-6ae2-4f09-a74d-1ff15927d692-istio-envoy" pod "cc159b5a-6ae2-4f09-a74d-1ff15927d692" (UID: "cc159b5a-6ae2-4f09-a74d-1ff15927d692"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:57:26.304415 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.304343    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc159b5a-6ae2-4f09-a74d-1ff15927d692-istio-podinfo" pod "cc159b5a-6ae2-4f09-a74d-1ff15927d692" (UID: "cc159b5a-6ae2-4f09-a74d-1ff15927d692"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 09:57:26.304500 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.304459    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc159b5a-6ae2-4f09-a74d-1ff15927d692-kube-api-access-hqc9j" pod "cc159b5a-6ae2-4f09-a74d-1ff15927d692" (UID: "cc159b5a-6ae2-4f09-a74d-1ff15927d692"). InnerVolumeSpecName "kube-api-access-hqc9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:57:26.403383 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.403357    4473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc159b5a-6ae2-4f09-a74d-1ff15927d692-istio-token\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:26.403383 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.403383    4473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc159b5a-6ae2-4f09-a74d-1ff15927d692-istio-podinfo\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:26.403563 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.403392    4473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc159b5a-6ae2-4f09-a74d-1ff15927d692-istio-envoy\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:26.403563 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.403401    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqc9j\" (UniqueName: \"kubernetes.io/projected/cc159b5a-6ae2-4f09-a74d-1ff15927d692-kube-api-access-hqc9j\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:26.522929 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.522894    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-910d50-7db8cfbf6c-w8z4h"]
Nov 03 09:57:26.531080 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:26.531050    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-910d50-7db8cfbf6c-w8z4h"]
Nov 03 09:57:27.061252 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.061135    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-98d5dd-d6c54bc97-xhlw4" podUID="600108c9-7889-4e9f-a213-dfe5e1be9ad6" containerName="payload-processing" containerID="cri-o://39ba029e47ebe74f4193fb5b8082e9d7de3126ee6ee3bf03c6241658795a4658" gracePeriod=25
Nov 03 09:57:27.110957 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.110930    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"
Nov 03 09:57:27.111095 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.110935    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr" event={"ID":"cc159b5a-6ae2-4f09-a74d-1ff15927d692","Type":"ContainerDied","Data":"791621d0a229dce644637e5ffa2a555b0548fda86ad062165bcc77064f661f0a"}
Nov 03 09:57:27.111095 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.111018    4473 scope.go:122] "RemoveContainer" containerID="cd9aeb3769f2c937b77f31625d243052b7d238938466e2a983af477c93f9141b"
Nov 03 09:57:27.170373 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.170342    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-98d5dd-77466f785f-drltw" podUID="576473d6-9fd5-4ce3-b85f-5053576c9de9" containerName="payload-pre-processing" containerID="cri-o://39e5fc439e5888131c7e7717eb2dd7facb92fa9bb51188921ba5ec6f8f34a9f5" gracePeriod=25
Nov 03 09:57:27.275695 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.275654    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx" podUID="8e84a474-def6-41dc-8516-5e5ccf42ff46" containerName="main" containerID="cri-o://def8668025fa79aaae2cca1fd592da75f74871183aad2ab4bbbf33264f79a5c0" gracePeriod=45
Nov 03 09:57:27.277461 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.277427    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6" podUID="25c366b3-b919-462b-b69a-82ce9dbd7436" containerName="main" containerID="cri-o://b59b9360a0b2d91ca4d5785943ab0bfe3c1900086465341d125d6f741bca7861" gracePeriod=45
Nov 03 09:57:27.330095 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.330023    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"]
Nov 03 09:57:27.377201 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:57:27.377166    4473 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25c366b3_b919_462b_b69a_82ce9dbd7436.slice/crio-conmon-b59b9360a0b2d91ca4d5785943ab0bfe3c1900086465341d125d6f741bca7861.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 09:57:27.381101 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.381073    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-77207e-openshift-default-69fb5d748f-wrkxr"]
Nov 03 09:57:27.412290 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.412247    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6" podUID="25c366b3-b919-462b-b69a-82ce9dbd7436" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": dial tcp 10.133.0.42:8000: connect: connection refused"
Nov 03 09:57:27.542408 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.542387    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx"
Nov 03 09:57:27.560514 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.560493    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6"
Nov 03 09:57:27.615886 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.615861    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e84a474-def6-41dc-8516-5e5ccf42ff46-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e84a474-def6-41dc-8516-5e5ccf42ff46-tls-certs\") pod \"8e84a474-def6-41dc-8516-5e5ccf42ff46\" (UID: \"8e84a474-def6-41dc-8516-5e5ccf42ff46\") "
Nov 03 09:57:27.616069 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.615920    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25c366b3-b919-462b-b69a-82ce9dbd7436-home\" (UniqueName: \"kubernetes.io/empty-dir/25c366b3-b919-462b-b69a-82ce9dbd7436-home\") pod \"25c366b3-b919-462b-b69a-82ce9dbd7436\" (UID: \"25c366b3-b919-462b-b69a-82ce9dbd7436\") "
Nov 03 09:57:27.616069 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.615988    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e84a474-def6-41dc-8516-5e5ccf42ff46-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e84a474-def6-41dc-8516-5e5ccf42ff46-dshm\") pod \"8e84a474-def6-41dc-8516-5e5ccf42ff46\" (UID: \"8e84a474-def6-41dc-8516-5e5ccf42ff46\") "
Nov 03 09:57:27.616069 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.616020    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25c366b3-b919-462b-b69a-82ce9dbd7436-dshm\" (UniqueName: \"kubernetes.io/empty-dir/25c366b3-b919-462b-b69a-82ce9dbd7436-dshm\") pod \"25c366b3-b919-462b-b69a-82ce9dbd7436\" (UID: \"25c366b3-b919-462b-b69a-82ce9dbd7436\") "
Nov 03 09:57:27.616069 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.616049    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e84a474-def6-41dc-8516-5e5ccf42ff46-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e84a474-def6-41dc-8516-5e5ccf42ff46-model-cache\") pod \"8e84a474-def6-41dc-8516-5e5ccf42ff46\" (UID: \"8e84a474-def6-41dc-8516-5e5ccf42ff46\") "
Nov 03 09:57:27.616332 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.616131    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25c366b3-b919-462b-b69a-82ce9dbd7436-tls-certs\" (UniqueName: \"kubernetes.io/secret/25c366b3-b919-462b-b69a-82ce9dbd7436-tls-certs\") pod \"25c366b3-b919-462b-b69a-82ce9dbd7436\" (UID: \"25c366b3-b919-462b-b69a-82ce9dbd7436\") "
Nov 03 09:57:27.616332 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.616160    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e84a474-def6-41dc-8516-5e5ccf42ff46-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e84a474-def6-41dc-8516-5e5ccf42ff46-tmp-dir\") pod \"8e84a474-def6-41dc-8516-5e5ccf42ff46\" (UID: \"8e84a474-def6-41dc-8516-5e5ccf42ff46\") "
Nov 03 09:57:27.616332 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.616193    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25c366b3-b919-462b-b69a-82ce9dbd7436-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25c366b3-b919-462b-b69a-82ce9dbd7436-model-cache\") pod \"25c366b3-b919-462b-b69a-82ce9dbd7436\" (UID: \"25c366b3-b919-462b-b69a-82ce9dbd7436\") "
Nov 03 09:57:27.616332 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.616201    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25c366b3-b919-462b-b69a-82ce9dbd7436-home" pod "25c366b3-b919-462b-b69a-82ce9dbd7436" (UID: "25c366b3-b919-462b-b69a-82ce9dbd7436"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:57:27.616332 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.616235    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e84a474-def6-41dc-8516-5e5ccf42ff46-home\" (UniqueName: \"kubernetes.io/empty-dir/8e84a474-def6-41dc-8516-5e5ccf42ff46-home\") pod \"8e84a474-def6-41dc-8516-5e5ccf42ff46\" (UID: \"8e84a474-def6-41dc-8516-5e5ccf42ff46\") "
Nov 03 09:57:27.616332 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.616271    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25c366b3-b919-462b-b69a-82ce9dbd7436-kube-api-access-mv8lx\" (UniqueName: \"kubernetes.io/projected/25c366b3-b919-462b-b69a-82ce9dbd7436-kube-api-access-mv8lx\") pod \"25c366b3-b919-462b-b69a-82ce9dbd7436\" (UID: \"25c366b3-b919-462b-b69a-82ce9dbd7436\") "
Nov 03 09:57:27.616332 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.616325    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25c366b3-b919-462b-b69a-82ce9dbd7436-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25c366b3-b919-462b-b69a-82ce9dbd7436-tmp-dir\") pod \"25c366b3-b919-462b-b69a-82ce9dbd7436\" (UID: \"25c366b3-b919-462b-b69a-82ce9dbd7436\") "
Nov 03 09:57:27.616738 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.616364    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e84a474-def6-41dc-8516-5e5ccf42ff46-kube-api-access-26jz6\" (UniqueName: \"kubernetes.io/projected/8e84a474-def6-41dc-8516-5e5ccf42ff46-kube-api-access-26jz6\") pod \"8e84a474-def6-41dc-8516-5e5ccf42ff46\" (UID: \"8e84a474-def6-41dc-8516-5e5ccf42ff46\") "
Nov 03 09:57:27.616738 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.616412    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e84a474-def6-41dc-8516-5e5ccf42ff46-tmp-dir" pod "8e84a474-def6-41dc-8516-5e5ccf42ff46" (UID: "8e84a474-def6-41dc-8516-5e5ccf42ff46"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:57:27.616738 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.616681    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e84a474-def6-41dc-8516-5e5ccf42ff46-home" pod "8e84a474-def6-41dc-8516-5e5ccf42ff46" (UID: "8e84a474-def6-41dc-8516-5e5ccf42ff46"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:57:27.616912 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.616832    4473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25c366b3-b919-462b-b69a-82ce9dbd7436-home\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:27.616912 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.616854    4473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e84a474-def6-41dc-8516-5e5ccf42ff46-tmp-dir\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:27.616912 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.616867    4473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e84a474-def6-41dc-8516-5e5ccf42ff46-home\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:27.616912 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.616875    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e84a474-def6-41dc-8516-5e5ccf42ff46-model-cache" pod "8e84a474-def6-41dc-8516-5e5ccf42ff46" (UID: "8e84a474-def6-41dc-8516-5e5ccf42ff46"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:57:27.617198 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.617086    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25c366b3-b919-462b-b69a-82ce9dbd7436-tmp-dir" pod "25c366b3-b919-462b-b69a-82ce9dbd7436" (UID: "25c366b3-b919-462b-b69a-82ce9dbd7436"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:57:27.618084 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.617465    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25c366b3-b919-462b-b69a-82ce9dbd7436-model-cache" pod "25c366b3-b919-462b-b69a-82ce9dbd7436" (UID: "25c366b3-b919-462b-b69a-82ce9dbd7436"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:57:27.619092 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.619066    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e84a474-def6-41dc-8516-5e5ccf42ff46-tls-certs" pod "8e84a474-def6-41dc-8516-5e5ccf42ff46" (UID: "8e84a474-def6-41dc-8516-5e5ccf42ff46"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:57:27.619244 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.619220    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25c366b3-b919-462b-b69a-82ce9dbd7436-tls-certs" pod "25c366b3-b919-462b-b69a-82ce9dbd7436" (UID: "25c366b3-b919-462b-b69a-82ce9dbd7436"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:57:27.619526 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.619492    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25c366b3-b919-462b-b69a-82ce9dbd7436-dshm" pod "25c366b3-b919-462b-b69a-82ce9dbd7436" (UID: "25c366b3-b919-462b-b69a-82ce9dbd7436"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:57:27.619844 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.619827    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e84a474-def6-41dc-8516-5e5ccf42ff46-dshm" pod "8e84a474-def6-41dc-8516-5e5ccf42ff46" (UID: "8e84a474-def6-41dc-8516-5e5ccf42ff46"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 09:57:27.620224 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.620203    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e84a474-def6-41dc-8516-5e5ccf42ff46-kube-api-access-26jz6" pod "8e84a474-def6-41dc-8516-5e5ccf42ff46" (UID: "8e84a474-def6-41dc-8516-5e5ccf42ff46"). InnerVolumeSpecName "kube-api-access-26jz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:57:27.620883 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.620865    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25c366b3-b919-462b-b69a-82ce9dbd7436-kube-api-access-mv8lx" pod "25c366b3-b919-462b-b69a-82ce9dbd7436" (UID: "25c366b3-b919-462b-b69a-82ce9dbd7436"). InnerVolumeSpecName "kube-api-access-mv8lx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:57:27.718231 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.718202    4473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25c366b3-b919-462b-b69a-82ce9dbd7436-tls-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:27.718231 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.718228    4473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25c366b3-b919-462b-b69a-82ce9dbd7436-model-cache\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:27.718468 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.718240    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv8lx\" (UniqueName: \"kubernetes.io/projected/25c366b3-b919-462b-b69a-82ce9dbd7436-kube-api-access-mv8lx\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:27.718468 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.718251    4473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25c366b3-b919-462b-b69a-82ce9dbd7436-tmp-dir\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:27.718468 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.718260    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26jz6\" (UniqueName: \"kubernetes.io/projected/8e84a474-def6-41dc-8516-5e5ccf42ff46-kube-api-access-26jz6\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:27.718468 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.718268    4473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e84a474-def6-41dc-8516-5e5ccf42ff46-tls-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:27.718468 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.718276    4473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e84a474-def6-41dc-8516-5e5ccf42ff46-dshm\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:27.718468 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.718285    4473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25c366b3-b919-462b-b69a-82ce9dbd7436-dshm\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:27.718468 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.718293    4473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e84a474-def6-41dc-8516-5e5ccf42ff46-model-cache\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:27.868719 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.868633    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="960d4e1b-7409-4a75-83f4-26289cb2b694" path="/var/lib/kubelet/pods/960d4e1b-7409-4a75-83f4-26289cb2b694/volumes"
Nov 03 09:57:27.869094 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:27.869079    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc159b5a-6ae2-4f09-a74d-1ff15927d692" path="/var/lib/kubelet/pods/cc159b5a-6ae2-4f09-a74d-1ff15927d692/volumes"
Nov 03 09:57:28.122290 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:28.122211    4473 generic.go:356] "Generic (PLEG): container finished" podID="25c366b3-b919-462b-b69a-82ce9dbd7436" containerID="b59b9360a0b2d91ca4d5785943ab0bfe3c1900086465341d125d6f741bca7861" exitCode=0
Nov 03 09:57:28.122290 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:28.122255    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6"
Nov 03 09:57:28.122872 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:28.122293    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6" event={"ID":"25c366b3-b919-462b-b69a-82ce9dbd7436","Type":"ContainerDied","Data":"b59b9360a0b2d91ca4d5785943ab0bfe3c1900086465341d125d6f741bca7861"}
Nov 03 09:57:28.122872 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:28.122329    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6" event={"ID":"25c366b3-b919-462b-b69a-82ce9dbd7436","Type":"ContainerDied","Data":"a21883144068b3e789bfa33f97081118438224c855a18ff096001606835d1e9b"}
Nov 03 09:57:28.122872 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:28.122347    4473 scope.go:122] "RemoveContainer" containerID="b59b9360a0b2d91ca4d5785943ab0bfe3c1900086465341d125d6f741bca7861"
Nov 03 09:57:28.123575 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:28.123554    4473 generic.go:356] "Generic (PLEG): container finished" podID="8e84a474-def6-41dc-8516-5e5ccf42ff46" containerID="def8668025fa79aaae2cca1fd592da75f74871183aad2ab4bbbf33264f79a5c0" exitCode=0
Nov 03 09:57:28.123637 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:28.123586    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx" event={"ID":"8e84a474-def6-41dc-8516-5e5ccf42ff46","Type":"ContainerDied","Data":"def8668025fa79aaae2cca1fd592da75f74871183aad2ab4bbbf33264f79a5c0"}
Nov 03 09:57:28.123637 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:28.123603    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx"
Nov 03 09:57:28.123734 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:28.123606    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx" event={"ID":"8e84a474-def6-41dc-8516-5e5ccf42ff46","Type":"ContainerDied","Data":"03d355bc945204293c3919215c0aa6e198d2d028e498b1b9b4a7732f8fdcfa6b"}
Nov 03 09:57:28.131401 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:28.131386    4473 scope.go:122] "RemoveContainer" containerID="b59b9360a0b2d91ca4d5785943ab0bfe3c1900086465341d125d6f741bca7861"
Nov 03 09:57:28.131633 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:57:28.131614    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b59b9360a0b2d91ca4d5785943ab0bfe3c1900086465341d125d6f741bca7861\": container with ID starting with b59b9360a0b2d91ca4d5785943ab0bfe3c1900086465341d125d6f741bca7861 not found: ID does not exist" containerID="b59b9360a0b2d91ca4d5785943ab0bfe3c1900086465341d125d6f741bca7861"
Nov 03 09:57:28.131773 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:28.131639    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b59b9360a0b2d91ca4d5785943ab0bfe3c1900086465341d125d6f741bca7861"} err="failed to get container status \"b59b9360a0b2d91ca4d5785943ab0bfe3c1900086465341d125d6f741bca7861\": rpc error: code = NotFound desc = could not find container \"b59b9360a0b2d91ca4d5785943ab0bfe3c1900086465341d125d6f741bca7861\": container with ID starting with b59b9360a0b2d91ca4d5785943ab0bfe3c1900086465341d125d6f741bca7861 not found: ID does not exist"
Nov 03 09:57:28.131773 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:28.131655    4473 scope.go:122] "RemoveContainer" containerID="def8668025fa79aaae2cca1fd592da75f74871183aad2ab4bbbf33264f79a5c0"
Nov 03 09:57:28.140100 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:28.140085    4473 scope.go:122] "RemoveContainer" containerID="def8668025fa79aaae2cca1fd592da75f74871183aad2ab4bbbf33264f79a5c0"
Nov 03 09:57:28.140371 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:57:28.140354    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"def8668025fa79aaae2cca1fd592da75f74871183aad2ab4bbbf33264f79a5c0\": container with ID starting with def8668025fa79aaae2cca1fd592da75f74871183aad2ab4bbbf33264f79a5c0 not found: ID does not exist" containerID="def8668025fa79aaae2cca1fd592da75f74871183aad2ab4bbbf33264f79a5c0"
Nov 03 09:57:28.140425 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:28.140378    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"def8668025fa79aaae2cca1fd592da75f74871183aad2ab4bbbf33264f79a5c0"} err="failed to get container status \"def8668025fa79aaae2cca1fd592da75f74871183aad2ab4bbbf33264f79a5c0\": rpc error: code = NotFound desc = could not find container \"def8668025fa79aaae2cca1fd592da75f74871183aad2ab4bbbf33264f79a5c0\": container with ID starting with def8668025fa79aaae2cca1fd592da75f74871183aad2ab4bbbf33264f79a5c0 not found: ID does not exist"
Nov 03 09:57:28.291369 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:28.291334    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6"]
Nov 03 09:57:28.298713 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:28.298690    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-98d5dd/premium-w3-98d5dd-kserve-c98b77cd9-xq2d6"]
Nov 03 09:57:28.414673 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:28.414601    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx"]
Nov 03 09:57:28.421790 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:28.421766    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-98d5dd/distinct-2-w3-98d5dd-kserve-78f95f768c-r92tx"]
Nov 03 09:57:29.872463 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:29.872411    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25c366b3-b919-462b-b69a-82ce9dbd7436" path="/var/lib/kubelet/pods/25c366b3-b919-462b-b69a-82ce9dbd7436/volumes"
Nov 03 09:57:29.873181 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:29.873155    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e84a474-def6-41dc-8516-5e5ccf42ff46" path="/var/lib/kubelet/pods/8e84a474-def6-41dc-8516-5e5ccf42ff46/volumes"
Nov 03 09:57:31.122767 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:31.122733    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:57:32.108292 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:32.108255    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-98d5dd-77466f785f-drltw" podUID="576473d6-9fd5-4ce3-b85f-5053576c9de9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:57:33.114601 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:33.114555    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-98d5dd-d6c54bc97-xhlw4" podUID="600108c9-7889-4e9f-a213-dfe5e1be9ad6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:57:39.106669 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:39.106630    4473 scope.go:122] "RemoveContainer" containerID="8fe90596c830a0795702f1c73d74da57d29f5d4650912193f9d9765534f9d132"
Nov 03 09:57:39.115415 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:39.115397    4473 scope.go:122] "RemoveContainer" containerID="1c3b07db48115e915e7c243dfb2c563bc0c7354e064ac32029490bfe731f4b17"
Nov 03 09:57:39.123446 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:39.123425    4473 scope.go:122] "RemoveContainer" containerID="931563f03ed88d4dbf35983cdafd7ef70b9ae6e364189e3de30d52d8a7de1d25"
Nov 03 09:57:42.108235 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:42.108199    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-98d5dd-77466f785f-drltw" podUID="576473d6-9fd5-4ce3-b85f-5053576c9de9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:57:43.114747 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:43.114705    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-98d5dd-d6c54bc97-xhlw4" podUID="600108c9-7889-4e9f-a213-dfe5e1be9ad6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 09:57:46.937467 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:57:46.937432    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:57:47.243514 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:47.243474    4473 generic.go:356] "Generic (PLEG): container finished" podID="600108c9-7889-4e9f-a213-dfe5e1be9ad6" containerID="39ba029e47ebe74f4193fb5b8082e9d7de3126ee6ee3bf03c6241658795a4658" exitCode=0
Nov 03 09:57:47.243701 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:47.243674    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-98d5dd-d6c54bc97-xhlw4" event={"ID":"600108c9-7889-4e9f-a213-dfe5e1be9ad6","Type":"ContainerDied","Data":"39ba029e47ebe74f4193fb5b8082e9d7de3126ee6ee3bf03c6241658795a4658"}
Nov 03 09:57:47.320537 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:47.320513    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-98d5dd-d6c54bc97-xhlw4"
Nov 03 09:57:47.417341 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:47.417319    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-98d5dd-77466f785f-drltw"
Nov 03 09:57:47.495861 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:47.495782    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/576473d6-9fd5-4ce3-b85f-5053576c9de9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/576473d6-9fd5-4ce3-b85f-5053576c9de9-plugins-config-volume\") pod \"576473d6-9fd5-4ce3-b85f-5053576c9de9\" (UID: \"576473d6-9fd5-4ce3-b85f-5053576c9de9\") "
Nov 03 09:57:47.495861 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:47.495856    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/600108c9-7889-4e9f-a213-dfe5e1be9ad6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/600108c9-7889-4e9f-a213-dfe5e1be9ad6-plugins-config-volume\") pod \"600108c9-7889-4e9f-a213-dfe5e1be9ad6\" (UID: \"600108c9-7889-4e9f-a213-dfe5e1be9ad6\") "
Nov 03 09:57:47.496130 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:47.495883    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/600108c9-7889-4e9f-a213-dfe5e1be9ad6-kube-api-access-vnxxp\" (UniqueName: \"kubernetes.io/projected/600108c9-7889-4e9f-a213-dfe5e1be9ad6-kube-api-access-vnxxp\") pod \"600108c9-7889-4e9f-a213-dfe5e1be9ad6\" (UID: \"600108c9-7889-4e9f-a213-dfe5e1be9ad6\") "
Nov 03 09:57:47.496130 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:47.495970    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/576473d6-9fd5-4ce3-b85f-5053576c9de9-kube-api-access-4szmc\" (UniqueName: \"kubernetes.io/projected/576473d6-9fd5-4ce3-b85f-5053576c9de9-kube-api-access-4szmc\") pod \"576473d6-9fd5-4ce3-b85f-5053576c9de9\" (UID: \"576473d6-9fd5-4ce3-b85f-5053576c9de9\") "
Nov 03 09:57:47.496266 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:47.496246    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/576473d6-9fd5-4ce3-b85f-5053576c9de9-plugins-config-volume" pod "576473d6-9fd5-4ce3-b85f-5053576c9de9" (UID: "576473d6-9fd5-4ce3-b85f-5053576c9de9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:57:47.496331 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:47.496298    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/600108c9-7889-4e9f-a213-dfe5e1be9ad6-plugins-config-volume" pod "600108c9-7889-4e9f-a213-dfe5e1be9ad6" (UID: "600108c9-7889-4e9f-a213-dfe5e1be9ad6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 09:57:47.498303 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:47.498280    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/600108c9-7889-4e9f-a213-dfe5e1be9ad6-kube-api-access-vnxxp" pod "600108c9-7889-4e9f-a213-dfe5e1be9ad6" (UID: "600108c9-7889-4e9f-a213-dfe5e1be9ad6"). InnerVolumeSpecName "kube-api-access-vnxxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:57:47.498418 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:47.498322    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/576473d6-9fd5-4ce3-b85f-5053576c9de9-kube-api-access-4szmc" pod "576473d6-9fd5-4ce3-b85f-5053576c9de9" (UID: "576473d6-9fd5-4ce3-b85f-5053576c9de9"). InnerVolumeSpecName "kube-api-access-4szmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:57:47.597274 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:47.597234    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4szmc\" (UniqueName: \"kubernetes.io/projected/576473d6-9fd5-4ce3-b85f-5053576c9de9-kube-api-access-4szmc\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:47.597274 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:47.597261    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/576473d6-9fd5-4ce3-b85f-5053576c9de9-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:47.597274 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:47.597270    4473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/600108c9-7889-4e9f-a213-dfe5e1be9ad6-plugins-config-volume\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:47.597274 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:47.597279    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnxxp\" (UniqueName: \"kubernetes.io/projected/600108c9-7889-4e9f-a213-dfe5e1be9ad6-kube-api-access-vnxxp\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:57:48.249979 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:48.249879    4473 generic.go:356] "Generic (PLEG): container finished" podID="576473d6-9fd5-4ce3-b85f-5053576c9de9" containerID="39e5fc439e5888131c7e7717eb2dd7facb92fa9bb51188921ba5ec6f8f34a9f5" exitCode=0
Nov 03 09:57:48.249979 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:48.249925    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-98d5dd-77466f785f-drltw"
Nov 03 09:57:48.249979 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:48.249970    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-98d5dd-77466f785f-drltw" event={"ID":"576473d6-9fd5-4ce3-b85f-5053576c9de9","Type":"ContainerDied","Data":"39e5fc439e5888131c7e7717eb2dd7facb92fa9bb51188921ba5ec6f8f34a9f5"}
Nov 03 09:57:48.250606 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:48.250021    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-98d5dd-77466f785f-drltw" event={"ID":"576473d6-9fd5-4ce3-b85f-5053576c9de9","Type":"ContainerDied","Data":"d8f64fa5504e5281a9403c03ebfd403097b922eb5509189934b767a1f7e56b43"}
Nov 03 09:57:48.250606 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:48.250045    4473 scope.go:122] "RemoveContainer" containerID="39e5fc439e5888131c7e7717eb2dd7facb92fa9bb51188921ba5ec6f8f34a9f5"
Nov 03 09:57:48.251476 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:48.251452    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-98d5dd-d6c54bc97-xhlw4" event={"ID":"600108c9-7889-4e9f-a213-dfe5e1be9ad6","Type":"ContainerDied","Data":"f22bad2a9dcbaf280f2a580b8c5e732cabd5e1dbf379ad949b3bbe8b9c9f3d34"}
Nov 03 09:57:48.251606 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:48.251497    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-98d5dd-d6c54bc97-xhlw4"
Nov 03 09:57:48.259285 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:48.259267    4473 scope.go:122] "RemoveContainer" containerID="39e5fc439e5888131c7e7717eb2dd7facb92fa9bb51188921ba5ec6f8f34a9f5"
Nov 03 09:57:48.259536 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:57:48.259515    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39e5fc439e5888131c7e7717eb2dd7facb92fa9bb51188921ba5ec6f8f34a9f5\": container with ID starting with 39e5fc439e5888131c7e7717eb2dd7facb92fa9bb51188921ba5ec6f8f34a9f5 not found: ID does not exist" containerID="39e5fc439e5888131c7e7717eb2dd7facb92fa9bb51188921ba5ec6f8f34a9f5"
Nov 03 09:57:48.259597 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:48.259544    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39e5fc439e5888131c7e7717eb2dd7facb92fa9bb51188921ba5ec6f8f34a9f5"} err="failed to get container status \"39e5fc439e5888131c7e7717eb2dd7facb92fa9bb51188921ba5ec6f8f34a9f5\": rpc error: code = NotFound desc = could not find container \"39e5fc439e5888131c7e7717eb2dd7facb92fa9bb51188921ba5ec6f8f34a9f5\": container with ID starting with 39e5fc439e5888131c7e7717eb2dd7facb92fa9bb51188921ba5ec6f8f34a9f5 not found: ID does not exist"
Nov 03 09:57:48.259597 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:48.259561    4473 scope.go:122] "RemoveContainer" containerID="39ba029e47ebe74f4193fb5b8082e9d7de3126ee6ee3bf03c6241658795a4658"
Nov 03 09:57:48.422122 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:48.422090    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-98d5dd-d6c54bc97-xhlw4"]
Nov 03 09:57:48.423965 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:48.423934    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-98d5dd-d6c54bc97-xhlw4"]
Nov 03 09:57:48.535471 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:48.535395    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-98d5dd-77466f785f-drltw"]
Nov 03 09:57:48.542837 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:48.542810    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-98d5dd-77466f785f-drltw"]
Nov 03 09:57:49.868430 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:49.868387    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="576473d6-9fd5-4ce3-b85f-5053576c9de9" path="/var/lib/kubelet/pods/576473d6-9fd5-4ce3-b85f-5053576c9de9/volumes"
Nov 03 09:57:49.868986 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:49.868913    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="600108c9-7889-4e9f-a213-dfe5e1be9ad6" path="/var/lib/kubelet/pods/600108c9-7889-4e9f-a213-dfe5e1be9ad6/volumes"
Nov 03 09:57:52.175130 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:52.175095    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:57:57.362509 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:57.362475    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-xbn6s_a025cb89-c80d-48f7-9f0a-93c8b6c24950/limitador/0.log"
Nov 03 09:57:59.542324 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:57:59.542291    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-6vvpk_73e8a24d-31cf-44f0-99a5-5d3bd7259449/manager/0.log"
Nov 03 09:58:10.580691 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:58:10.580651    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-p72nd_dc80f365-2a40-4fd0-ae2b-6cfc815a3d63/proxy/0.log"
Nov 03 09:58:11.363363 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:58:11.363329    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7dd5f74f7b-lk7c9_17e69a7f-4e64-4aad-b798-af8fa74a2b76/manager/0.log"
Nov 03 09:58:16.936731 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:58:16.936650    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:58:34.245923 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:58:34.245886    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:58:37.052467 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:58:37.052427    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af909b33-5cnmx"]
Nov 03 09:58:37.103022 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:58:37.102990    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af909b33-5cnmx"]
Nov 03 09:58:37.867601 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:58:37.867571    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1464a718-105e-4039-b0be-b7955701e38d" path="/var/lib/kubelet/pods/1464a718-105e-4039-b0be-b7955701e38d/volumes"
Nov 03 09:58:39.184276 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:58:39.184244    4473 scope.go:122] "RemoveContainer" containerID="e9cf14080c5252e7607e780f54ae19a87aa914dcce39e6de84b62da48b86f84e"
Nov 03 09:58:39.193195 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:58:39.193175    4473 scope.go:122] "RemoveContainer" containerID="e67de5123a218acada7e208b9456c5d53c053b05c96d8fa03a0f3518a70d1c23"
Nov 03 09:58:39.201520 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:58:39.201498    4473 scope.go:122] "RemoveContainer" containerID="25d57005b8aee56f4ad1d1f922e85620f1360e452bf227f02e7df6d1cc0ee443"
Nov 03 09:58:46.936756 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:58:46.936716    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:58:47.052612 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:58:47.052577    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d8d638a-zkz5t"]
Nov 03 09:58:47.102669 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:58:47.102643    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d8d638a-zkz5t"]
Nov 03 09:58:47.868556 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:58:47.868520    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e1c80a4-7dd5-4341-b7a6-b534d026b54c" path="/var/lib/kubelet/pods/5e1c80a4-7dd5-4341-b7a6-b534d026b54c/volumes"
Nov 03 09:58:49.060146 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:58:49.060112    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df73bc75-zl9lt"]
Nov 03 09:58:49.110275 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:58:49.110245    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df73bc75-zl9lt"]
Nov 03 09:58:49.868300 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:58:49.868268    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e87c013-ff41-4774-8e96-838df05ecc08" path="/var/lib/kubelet/pods/3e87c013-ff41-4774-8e96-838df05ecc08/volumes"
Nov 03 09:58:59.057080 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:58:59.057042    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7123bf8-s4l9j"]
Nov 03 09:58:59.107141 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:58:59.107114    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7123bf8-s4l9j"]
Nov 03 09:58:59.868513 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:58:59.868478    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6893d9f-cd25-4ded-afa9-c3e226ea9f9a" path="/var/lib/kubelet/pods/b6893d9f-cd25-4ded-afa9-c3e226ea9f9a/volumes"
Nov 03 09:59:15.053581 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:15.053546    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2093da2-wdlxd"]
Nov 03 09:59:15.103967 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:15.103919    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2093da2-wdlxd"]
Nov 03 09:59:15.868958 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:15.868915    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08906c75-24d6-4cdc-8220-f93ca8d85d7a" path="/var/lib/kubelet/pods/08906c75-24d6-4cdc-8220-f93ca8d85d7a/volumes"
Nov 03 09:59:16.936483 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:59:16.936451    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:59:20.882313 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:20.882278    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 09:59:26.051793 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:26.051754    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-750cbe2e-kgmtv"]
Nov 03 09:59:26.102053 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:26.102018    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-750cbe2e-kgmtv"]
Nov 03 09:59:26.544533 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:26.544499    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7dd5f74f7b-lk7c9"]
Nov 03 09:59:26.544806 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:26.544781    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7dd5f74f7b-lk7c9" podUID="17e69a7f-4e64-4aad-b798-af8fa74a2b76" containerName="manager" containerID="cri-o://339a02004ef32a7c64014efdcd89a49bf9a2b6f267e85d31d748f42e98b1ffa4" gracePeriod=30
Nov 03 09:59:26.744895 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:26.744863    4473 generic.go:356] "Generic (PLEG): container finished" podID="17e69a7f-4e64-4aad-b798-af8fa74a2b76" containerID="339a02004ef32a7c64014efdcd89a49bf9a2b6f267e85d31d748f42e98b1ffa4" exitCode=0
Nov 03 09:59:26.745068 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:26.744938    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dd5f74f7b-lk7c9" event={"ID":"17e69a7f-4e64-4aad-b798-af8fa74a2b76","Type":"ContainerDied","Data":"339a02004ef32a7c64014efdcd89a49bf9a2b6f267e85d31d748f42e98b1ffa4"}
Nov 03 09:59:26.745115 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:26.745036    4473 scope.go:122] "RemoveContainer" containerID="367fb63de7da5a3a8d40ca0f481177808f079ee612f30457884b3c459ae49741"
Nov 03 09:59:26.785320 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:26.785301    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dd5f74f7b-lk7c9"
Nov 03 09:59:26.882455 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:26.882424    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17e69a7f-4e64-4aad-b798-af8fa74a2b76-webhook-cert\" (UniqueName: \"kubernetes.io/secret/17e69a7f-4e64-4aad-b798-af8fa74a2b76-webhook-cert\") pod \"17e69a7f-4e64-4aad-b798-af8fa74a2b76\" (UID: \"17e69a7f-4e64-4aad-b798-af8fa74a2b76\") "
Nov 03 09:59:26.882617 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:26.882465    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17e69a7f-4e64-4aad-b798-af8fa74a2b76-metrics-certs\" (UniqueName: \"kubernetes.io/secret/17e69a7f-4e64-4aad-b798-af8fa74a2b76-metrics-certs\") pod \"17e69a7f-4e64-4aad-b798-af8fa74a2b76\" (UID: \"17e69a7f-4e64-4aad-b798-af8fa74a2b76\") "
Nov 03 09:59:26.882617 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:26.882546    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17e69a7f-4e64-4aad-b798-af8fa74a2b76-kube-api-access-pp5m5\" (UniqueName: \"kubernetes.io/projected/17e69a7f-4e64-4aad-b798-af8fa74a2b76-kube-api-access-pp5m5\") pod \"17e69a7f-4e64-4aad-b798-af8fa74a2b76\" (UID: \"17e69a7f-4e64-4aad-b798-af8fa74a2b76\") "
Nov 03 09:59:26.884755 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:26.884726    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17e69a7f-4e64-4aad-b798-af8fa74a2b76-metrics-certs" pod "17e69a7f-4e64-4aad-b798-af8fa74a2b76" (UID: "17e69a7f-4e64-4aad-b798-af8fa74a2b76"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:59:26.884875 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:26.884793    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e69a7f-4e64-4aad-b798-af8fa74a2b76-kube-api-access-pp5m5" pod "17e69a7f-4e64-4aad-b798-af8fa74a2b76" (UID: "17e69a7f-4e64-4aad-b798-af8fa74a2b76"). InnerVolumeSpecName "kube-api-access-pp5m5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 09:59:26.884875 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:26.884805    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17e69a7f-4e64-4aad-b798-af8fa74a2b76-webhook-cert" pod "17e69a7f-4e64-4aad-b798-af8fa74a2b76" (UID: "17e69a7f-4e64-4aad-b798-af8fa74a2b76"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 09:59:26.983686 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:26.983650    4473 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/17e69a7f-4e64-4aad-b798-af8fa74a2b76-webhook-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:59:26.983686 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:26.983679    4473 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17e69a7f-4e64-4aad-b798-af8fa74a2b76-metrics-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:59:26.983893 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:26.983692    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp5m5\" (UniqueName: \"kubernetes.io/projected/17e69a7f-4e64-4aad-b798-af8fa74a2b76-kube-api-access-pp5m5\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 09:59:27.751745 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:27.751704    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dd5f74f7b-lk7c9" event={"ID":"17e69a7f-4e64-4aad-b798-af8fa74a2b76","Type":"ContainerDied","Data":"c26f3e46f41fe04330875db625dd78e966fccc824975db918465880400e702c0"}
Nov 03 09:59:27.752477 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:27.751717    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dd5f74f7b-lk7c9"
Nov 03 09:59:27.752477 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:27.751823    4473 scope.go:122] "RemoveContainer" containerID="339a02004ef32a7c64014efdcd89a49bf9a2b6f267e85d31d748f42e98b1ffa4"
Nov 03 09:59:27.802256 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:59:27.802227    4473 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e69a7f_4e64_4aad_b798_af8fa74a2b76.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e69a7f_4e64_4aad_b798_af8fa74a2b76.slice/crio-c26f3e46f41fe04330875db625dd78e966fccc824975db918465880400e702c0\": RecentStats: unable to find data in memory cache]"
Nov 03 09:59:27.869681 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:27.869645    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73981a22-201c-42d3-b1d6-81afe3c9d24f" path="/var/lib/kubelet/pods/73981a22-201c-42d3-b1d6-81afe3c9d24f/volumes"
Nov 03 09:59:27.919491 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:27.919454    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7dd5f74f7b-lk7c9"]
Nov 03 09:59:27.969908 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:27.969878    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7dd5f74f7b-lk7c9"]
Nov 03 09:59:28.050395 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:28.050327    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07d668f1-k49m4"]
Nov 03 09:59:28.055736 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:28.055711    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07d668f1-k49m4"]
Nov 03 09:59:29.868140 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:29.868109    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17e69a7f-4e64-4aad-b798-af8fa74a2b76" path="/var/lib/kubelet/pods/17e69a7f-4e64-4aad-b798-af8fa74a2b76/volumes"
Nov 03 09:59:29.868511 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:29.868497    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89f277d9-a48e-4fbe-b866-b26ef10468d4" path="/var/lib/kubelet/pods/89f277d9-a48e-4fbe-b866-b26ef10468d4/volumes"
Nov 03 09:59:36.053881 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:36.053849    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56e07e4a-lc726"]
Nov 03 09:59:36.104010 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:36.103979    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56e07e4a-lc726"]
Nov 03 09:59:36.904714 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:36.904677    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 09:59:36.904902 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:36.904749    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 09:59:37.869038 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:37.868999    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45e6234a-28a0-47ff-a266-815869ef9666" path="/var/lib/kubelet/pods/45e6234a-28a0-47ff-a266-815869ef9666/volumes"
Nov 03 09:59:39.238408 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:39.238376    4473 scope.go:122] "RemoveContainer" containerID="57494d409a83b6be5506740f548a47da39160bc7800b5cd7e52f493777b33a95"
Nov 03 09:59:46.936500 ip-10-0-139-73 kubenswrapper[4473]: E1103 09:59:46.936415    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 09:59:48.064484 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:48.064449    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc1db4ce-6hvxg"]
Nov 03 09:59:48.114537 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:48.114505    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc1db4ce-6hvxg"]
Nov 03 09:59:49.868717 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:49.868667    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55d63835-33f6-4b7d-8d7d-894ebc43a641" path="/var/lib/kubelet/pods/55d63835-33f6-4b7d-8d7d-894ebc43a641/volumes"
Nov 03 09:59:59.051133 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:59.051102    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57788ad7-p2m5c"]
Nov 03 09:59:59.101236 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:59.101211    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57788ad7-p2m5c"]
Nov 03 09:59:59.868784 ip-10-0-139-73 kubenswrapper[4473]: I1103 09:59:59.868752    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5605651d-4be1-4b2c-a4a3-827f6fd13146" path="/var/lib/kubelet/pods/5605651d-4be1-4b2c-a4a3-827f6fd13146/volumes"
Nov 03 10:00:16.935553 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:00:16.935517    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:00:31.245569 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:00:31.245538    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:00:39.264051 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:00:39.264013    4473 scope.go:122] "RemoveContainer" containerID="d7054a28bca74f4cffb3f81bb236acd7e809353af3816e823bdd8122c8344bb9"
Nov 03 10:00:39.272527 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:00:39.272503    4473 scope.go:122] "RemoveContainer" containerID="59c404aa6d80b483b04967dfe75b8de0ede492fabd07f0d047d6f157dc0375a4"
Nov 03 10:00:39.282729 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:00:39.282704    4473 scope.go:122] "RemoveContainer" containerID="8af805c6572d7a5528eec711875db25d1a976847c5babc96fa8ad02ced3fe548"
Nov 03 10:00:39.290575 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:00:39.290554    4473 scope.go:122] "RemoveContainer" containerID="03af178a474f1198f9eb70fc2b6c9d901e182b867b59547f1749245c95933263"
Nov 03 10:00:39.298013 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:00:39.297987    4473 scope.go:122] "RemoveContainer" containerID="f2f1da200b6e3038534a4f417a87f0512f72cd5f0243d9f841511fd97590de0f"
Nov 03 10:00:39.305351 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:00:39.305333    4473 scope.go:122] "RemoveContainer" containerID="b54b6a7e98fc2cda172f57494563ee392e2c7a4a548964d7974fb0afb6039986"
Nov 03 10:00:39.312561 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:00:39.312543    4473 scope.go:122] "RemoveContainer" containerID="92f683f583b52c9933e2a325bbfd42b183218fe711e8a5f758e91ae962c110e7"
Nov 03 10:00:46.936266 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:00:46.936232    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:00:49.670095 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:00:49.670065    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:00:59.161464 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:00:59.161430    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:01:02.302693 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.302656    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"]
Nov 03 10:01:02.303371 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.303204    4473 cpu_manager.go:414] "containerMap: removing container" podUID="d8582e32-8742-478d-9cf8-67a8488ab67c" containerName="revoke-keys"
Nov 03 10:01:02.303371 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.303224    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8582e32-8742-478d-9cf8-67a8488ab67c" containerName="revoke-keys"
Nov 03 10:01:02.303371 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.303244    4473 cpu_manager.go:414] "containerMap: removing container" podUID="600108c9-7889-4e9f-a213-dfe5e1be9ad6" containerName="payload-processing"
Nov 03 10:01:02.303371 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.303252    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="600108c9-7889-4e9f-a213-dfe5e1be9ad6" containerName="payload-processing"
Nov 03 10:01:02.303371 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.303269    4473 cpu_manager.go:414] "containerMap: removing container" podUID="a697a58e-ba2e-4bff-a9fb-0d749cc6e044" containerName="istio-proxy"
Nov 03 10:01:02.303371 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.303276    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a697a58e-ba2e-4bff-a9fb-0d749cc6e044" containerName="istio-proxy"
Nov 03 10:01:02.303371 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.303286    4473 cpu_manager.go:414] "containerMap: removing container" podUID="960d4e1b-7409-4a75-83f4-26289cb2b694" containerName="payload-pre-processing"
Nov 03 10:01:02.303371 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.303293    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="960d4e1b-7409-4a75-83f4-26289cb2b694" containerName="payload-pre-processing"
Nov 03 10:01:02.303371 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.303311    4473 cpu_manager.go:414] "containerMap: removing container" podUID="035b0461-2dfa-42f4-8ea7-2d1911cddca9" containerName="main"
Nov 03 10:01:02.303371 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.303318    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="035b0461-2dfa-42f4-8ea7-2d1911cddca9" containerName="main"
Nov 03 10:01:02.303371 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.303332    4473 cpu_manager.go:414] "containerMap: removing container" podUID="576473d6-9fd5-4ce3-b85f-5053576c9de9" containerName="payload-pre-processing"
Nov 03 10:01:02.303371 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.303340    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="576473d6-9fd5-4ce3-b85f-5053576c9de9" containerName="payload-pre-processing"
Nov 03 10:01:02.303371 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.303350    4473 cpu_manager.go:414] "containerMap: removing container" podUID="17e69a7f-4e64-4aad-b798-af8fa74a2b76" containerName="manager"
Nov 03 10:01:02.303371 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.303357    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17e69a7f-4e64-4aad-b798-af8fa74a2b76" containerName="manager"
Nov 03 10:01:02.303371 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.303374    4473 cpu_manager.go:414] "containerMap: removing container" podUID="6afd6431-8c5e-4a46-b505-270efaaacd1c" containerName="main"
Nov 03 10:01:02.303371 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.303380    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6afd6431-8c5e-4a46-b505-270efaaacd1c" containerName="main"
Nov 03 10:01:02.304337 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.303390    4473 cpu_manager.go:414] "containerMap: removing container" podUID="cc159b5a-6ae2-4f09-a74d-1ff15927d692" containerName="istio-proxy"
Nov 03 10:01:02.304337 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.303396    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc159b5a-6ae2-4f09-a74d-1ff15927d692" containerName="istio-proxy"
Nov 03 10:01:02.304337 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.303412    4473 cpu_manager.go:414] "containerMap: removing container" podUID="25c366b3-b919-462b-b69a-82ce9dbd7436" containerName="main"
Nov 03 10:01:02.304337 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.303419    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25c366b3-b919-462b-b69a-82ce9dbd7436" containerName="main"
Nov 03 10:01:02.304337 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.303435    4473 cpu_manager.go:414] "containerMap: removing container" podUID="8e84a474-def6-41dc-8516-5e5ccf42ff46" containerName="main"
Nov 03 10:01:02.304337 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.303442    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e84a474-def6-41dc-8516-5e5ccf42ff46" containerName="main"
Nov 03 10:01:02.304337 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.303518    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="6afd6431-8c5e-4a46-b505-270efaaacd1c" containerName="main"
Nov 03 10:01:02.304337 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.303531    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc159b5a-6ae2-4f09-a74d-1ff15927d692" containerName="istio-proxy"
Nov 03 10:01:02.304337 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.303541    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="25c366b3-b919-462b-b69a-82ce9dbd7436" containerName="main"
Nov 03 10:01:02.304337 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.303549    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="17e69a7f-4e64-4aad-b798-af8fa74a2b76" containerName="manager"
Nov 03 10:01:02.304337 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.303560    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e84a474-def6-41dc-8516-5e5ccf42ff46" containerName="main"
Nov 03 10:01:02.304337 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.303570    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8582e32-8742-478d-9cf8-67a8488ab67c" containerName="revoke-keys"
Nov 03 10:01:02.304337 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.303577    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="600108c9-7889-4e9f-a213-dfe5e1be9ad6" containerName="payload-processing"
Nov 03 10:01:02.304337 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.303587    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="a697a58e-ba2e-4bff-a9fb-0d749cc6e044" containerName="istio-proxy"
Nov 03 10:01:02.304337 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.303594    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="960d4e1b-7409-4a75-83f4-26289cb2b694" containerName="payload-pre-processing"
Nov 03 10:01:02.304337 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.303601    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="035b0461-2dfa-42f4-8ea7-2d1911cddca9" containerName="main"
Nov 03 10:01:02.304337 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.303609    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="576473d6-9fd5-4ce3-b85f-5053576c9de9" containerName="payload-pre-processing"
Nov 03 10:01:02.306867 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.306847    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"
Nov 03 10:01:02.393315 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.393279    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"]
Nov 03 10:01:02.492479 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.492450    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea5d-0246-4fe2-8dcb-3363626c6018-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c7mlq\" (UID: \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"
Nov 03 10:01:02.492479 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.492484    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb74ea5d-0246-4fe2-8dcb-3363626c6018-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c7mlq\" (UID: \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"
Nov 03 10:01:02.492766 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.492552    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea5d-0246-4fe2-8dcb-3363626c6018-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c7mlq\" (UID: \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"
Nov 03 10:01:02.492766 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.492674    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb74ea5d-0246-4fe2-8dcb-3363626c6018-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c7mlq\" (UID: \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"
Nov 03 10:01:02.492766 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.492751    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea5d-0246-4fe2-8dcb-3363626c6018-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c7mlq\" (UID: \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"
Nov 03 10:01:02.492982 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.492807    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea5d-0246-4fe2-8dcb-3363626c6018-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c7mlq\" (UID: \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"
Nov 03 10:01:02.492982 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.492850    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea5d-0246-4fe2-8dcb-3363626c6018-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c7mlq\" (UID: \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"
Nov 03 10:01:02.492982 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.492878    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dfqv\" (UniqueName: \"kubernetes.io/projected/fb74ea5d-0246-4fe2-8dcb-3363626c6018-kube-api-access-9dfqv\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c7mlq\" (UID: \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"
Nov 03 10:01:02.492982 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.492907    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb74ea5d-0246-4fe2-8dcb-3363626c6018-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c7mlq\" (UID: \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"
Nov 03 10:01:02.594386 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.594308    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea5d-0246-4fe2-8dcb-3363626c6018-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c7mlq\" (UID: \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"
Nov 03 10:01:02.594386 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.594350    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea5d-0246-4fe2-8dcb-3363626c6018-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c7mlq\" (UID: \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"
Nov 03 10:01:02.594386 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.594375    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea5d-0246-4fe2-8dcb-3363626c6018-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c7mlq\" (UID: \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"
Nov 03 10:01:02.594661 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.594393    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dfqv\" (UniqueName: \"kubernetes.io/projected/fb74ea5d-0246-4fe2-8dcb-3363626c6018-kube-api-access-9dfqv\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c7mlq\" (UID: \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"
Nov 03 10:01:02.594661 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.594414    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb74ea5d-0246-4fe2-8dcb-3363626c6018-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c7mlq\" (UID: \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"
Nov 03 10:01:02.594661 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.594471    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea5d-0246-4fe2-8dcb-3363626c6018-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c7mlq\" (UID: \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"
Nov 03 10:01:02.594661 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.594499    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb74ea5d-0246-4fe2-8dcb-3363626c6018-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c7mlq\" (UID: \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"
Nov 03 10:01:02.594661 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.594563    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea5d-0246-4fe2-8dcb-3363626c6018-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c7mlq\" (UID: \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"
Nov 03 10:01:02.594661 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.594587    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb74ea5d-0246-4fe2-8dcb-3363626c6018-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c7mlq\" (UID: \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"
Nov 03 10:01:02.595048 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.594749    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea5d-0246-4fe2-8dcb-3363626c6018-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c7mlq\" (UID: \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"
Nov 03 10:01:02.595048 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.594788    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea5d-0246-4fe2-8dcb-3363626c6018-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c7mlq\" (UID: \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"
Nov 03 10:01:02.595176 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.595041    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea5d-0246-4fe2-8dcb-3363626c6018-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c7mlq\" (UID: \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"
Nov 03 10:01:02.595176 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.595088    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea5d-0246-4fe2-8dcb-3363626c6018-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c7mlq\" (UID: \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"
Nov 03 10:01:02.595370 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.595351    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb74ea5d-0246-4fe2-8dcb-3363626c6018-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c7mlq\" (UID: \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"
Nov 03 10:01:02.596927 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.596899    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea5d-0246-4fe2-8dcb-3363626c6018-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c7mlq\" (UID: \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"
Nov 03 10:01:02.597252 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.597232    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb74ea5d-0246-4fe2-8dcb-3363626c6018-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c7mlq\" (UID: \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"
Nov 03 10:01:02.653740 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.653709    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb74ea5d-0246-4fe2-8dcb-3363626c6018-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c7mlq\" (UID: \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"
Nov 03 10:01:02.653868 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.653786    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dfqv\" (UniqueName: \"kubernetes.io/projected/fb74ea5d-0246-4fe2-8dcb-3363626c6018-kube-api-access-9dfqv\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-c7mlq\" (UID: \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"
Nov 03 10:01:02.918662 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:02.918632    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"
Nov 03 10:01:03.343787 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:03.343764    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"]
Nov 03 10:01:03.345810 ip-10-0-139-73 kubenswrapper[4473]: W1103 10:01:03.345778    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb74ea5d_0246_4fe2_8dcb_3363626c6018.slice/crio-7f3ec5e6ee5233280677bcdb452068171b8b659bda8d349ee5d4906fcd307703 WatchSource:0}: Error finding container 7f3ec5e6ee5233280677bcdb452068171b8b659bda8d349ee5d4906fcd307703: Status 404 returned error can't find the container with id 7f3ec5e6ee5233280677bcdb452068171b8b659bda8d349ee5d4906fcd307703
Nov 03 10:01:03.347880 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:03.347852    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:01:03.347982 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:03.347915    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:01:03.347982 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:03.347968    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:01:04.203439 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:04.203402    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq" event={"ID":"fb74ea5d-0246-4fe2-8dcb-3363626c6018","Type":"ContainerStarted","Data":"68c7cc705b268144f5d8c1d337e4d099257f0bc95573461fbc9ef043101602ed"}
Nov 03 10:01:04.203668 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:04.203470    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq" event={"ID":"fb74ea5d-0246-4fe2-8dcb-3363626c6018","Type":"ContainerStarted","Data":"7f3ec5e6ee5233280677bcdb452068171b8b659bda8d349ee5d4906fcd307703"}
Nov 03 10:01:04.375018 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:04.374933    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq" podStartSLOduration=2.37491486 podStartE2EDuration="2.37491486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:01:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:01:04.371521948 +0000 UTC m=+4888.640585421" watchObservedRunningTime="2026-11-03 10:01:04.37491486 +0000 UTC m=+4888.643978295"
Nov 03 10:01:04.921602 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:04.921530    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"
Nov 03 10:01:04.922134 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:04.922050    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Nov 03 10:01:04.922134 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:04.922110    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq" podUID="fb74ea5d-0246-4fe2-8dcb-3363626c6018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Nov 03 10:01:05.933338 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:05.933295    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Nov 03 10:01:05.933844 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:05.933360    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq" podUID="fb74ea5d-0246-4fe2-8dcb-3363626c6018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Nov 03 10:01:06.927971 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:06.925360    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Nov 03 10:01:06.927971 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:06.925421    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq" podUID="fb74ea5d-0246-4fe2-8dcb-3363626c6018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Nov 03 10:01:07.937403 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:07.937363    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Nov 03 10:01:07.938036 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:07.938004    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq" podUID="fb74ea5d-0246-4fe2-8dcb-3363626c6018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Nov 03 10:01:08.941095 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:08.941056    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Nov 03 10:01:08.941511 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:08.941124    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq" podUID="fb74ea5d-0246-4fe2-8dcb-3363626c6018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Nov 03 10:01:09.461928 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:09.461895    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:01:09.919326 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:09.919287    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused" start-of-body=
Nov 03 10:01:09.919541 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:09.919360    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq" podUID="fb74ea5d-0246-4fe2-8dcb-3363626c6018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": dial tcp 10.133.0.252:15021: connect: connection refused"
Nov 03 10:01:11.920146 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:11.920114    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:01:11.920610 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:11.920253    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq" podUID="fb74ea5d-0246-4fe2-8dcb-3363626c6018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:01:12.962152 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:12.962109    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:01:12.962631 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:12.962176    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq" podUID="fb74ea5d-0246-4fe2-8dcb-3363626c6018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:01:13.719735 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:13.719698    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"
Nov 03 10:01:13.720189 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:13.720167    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"
Nov 03 10:01:13.720901 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:13.720885    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"
Nov 03 10:01:16.936788 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:01:16.936694    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:01:17.409555 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:17.409523    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"]
Nov 03 10:01:17.409910 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:17.409897    4473 cpu_manager.go:414] "containerMap: removing container" podUID="17e69a7f-4e64-4aad-b798-af8fa74a2b76" containerName="manager"
Nov 03 10:01:17.409983 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:17.409912    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17e69a7f-4e64-4aad-b798-af8fa74a2b76" containerName="manager"
Nov 03 10:01:17.410035 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:17.410023    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="17e69a7f-4e64-4aad-b798-af8fa74a2b76" containerName="manager"
Nov 03 10:01:17.413291 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:17.413275    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"
Nov 03 10:01:17.444485 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:17.444459    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/031e6c9b-93f8-458b-a20d-24927d6fb383-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9wsvs\" (UID: \"031e6c9b-93f8-458b-a20d-24927d6fb383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"
Nov 03 10:01:17.444618 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:17.444490    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/031e6c9b-93f8-458b-a20d-24927d6fb383-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9wsvs\" (UID: \"031e6c9b-93f8-458b-a20d-24927d6fb383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"
Nov 03 10:01:17.444618 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:17.444523    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/031e6c9b-93f8-458b-a20d-24927d6fb383-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9wsvs\" (UID: \"031e6c9b-93f8-458b-a20d-24927d6fb383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"
Nov 03 10:01:17.444764 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:17.444670    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/031e6c9b-93f8-458b-a20d-24927d6fb383-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9wsvs\" (UID: \"031e6c9b-93f8-458b-a20d-24927d6fb383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"
Nov 03 10:01:17.444764 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:17.444716    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2jtg\" (UniqueName: \"kubernetes.io/projected/031e6c9b-93f8-458b-a20d-24927d6fb383-kube-api-access-b2jtg\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9wsvs\" (UID: \"031e6c9b-93f8-458b-a20d-24927d6fb383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"
Nov 03 10:01:17.444874 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:17.444784    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/031e6c9b-93f8-458b-a20d-24927d6fb383-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9wsvs\" (UID: \"031e6c9b-93f8-458b-a20d-24927d6fb383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"
Nov 03 10:01:17.444874 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:17.444850    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/031e6c9b-93f8-458b-a20d-24927d6fb383-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9wsvs\" (UID: \"031e6c9b-93f8-458b-a20d-24927d6fb383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"
Nov 03 10:01:17.444982 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:17.444884    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/031e6c9b-93f8-458b-a20d-24927d6fb383-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9wsvs\" (UID: \"031e6c9b-93f8-458b-a20d-24927d6fb383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"
Nov 03 10:01:17.444982 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:17.444903    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/031e6c9b-93f8-458b-a20d-24927d6fb383-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9wsvs\" (UID: \"031e6c9b-93f8-458b-a20d-24927d6fb383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"
Nov 03 10:01:17.518979 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:17.518932    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"]
Nov 03 10:01:17.546162 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:17.546130    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/031e6c9b-93f8-458b-a20d-24927d6fb383-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9wsvs\" (UID: \"031e6c9b-93f8-458b-a20d-24927d6fb383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"
Nov 03 10:01:17.546327 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:17.546167    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2jtg\" (UniqueName: \"kubernetes.io/projected/031e6c9b-93f8-458b-a20d-24927d6fb383-kube-api-access-b2jtg\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9wsvs\" (UID: \"031e6c9b-93f8-458b-a20d-24927d6fb383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"
Nov 03 10:01:17.546327 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:17.546295    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/031e6c9b-93f8-458b-a20d-24927d6fb383-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9wsvs\" (UID: \"031e6c9b-93f8-458b-a20d-24927d6fb383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"
Nov 03 10:01:17.546469 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:17.546349    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/031e6c9b-93f8-458b-a20d-24927d6fb383-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9wsvs\" (UID: \"031e6c9b-93f8-458b-a20d-24927d6fb383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"
Nov 03 10:01:17.546469 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:17.546390    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/031e6c9b-93f8-458b-a20d-24927d6fb383-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9wsvs\" (UID: \"031e6c9b-93f8-458b-a20d-24927d6fb383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"
Nov 03 10:01:17.546469 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:17.546419    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/031e6c9b-93f8-458b-a20d-24927d6fb383-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9wsvs\" (UID: \"031e6c9b-93f8-458b-a20d-24927d6fb383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"
Nov 03 10:01:17.546615 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:17.546502    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/031e6c9b-93f8-458b-a20d-24927d6fb383-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9wsvs\" (UID: \"031e6c9b-93f8-458b-a20d-24927d6fb383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"
Nov 03 10:01:17.546615 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:17.546514    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/031e6c9b-93f8-458b-a20d-24927d6fb383-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9wsvs\" (UID: \"031e6c9b-93f8-458b-a20d-24927d6fb383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"
Nov 03 10:01:17.546615 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:17.546531    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/031e6c9b-93f8-458b-a20d-24927d6fb383-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9wsvs\" (UID: \"031e6c9b-93f8-458b-a20d-24927d6fb383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"
Nov 03 10:01:17.546756 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:17.546616    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/031e6c9b-93f8-458b-a20d-24927d6fb383-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9wsvs\" (UID: \"031e6c9b-93f8-458b-a20d-24927d6fb383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"
Nov 03 10:01:17.546756 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:17.546655    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/031e6c9b-93f8-458b-a20d-24927d6fb383-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9wsvs\" (UID: \"031e6c9b-93f8-458b-a20d-24927d6fb383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"
Nov 03 10:01:17.546988 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:17.546964    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/031e6c9b-93f8-458b-a20d-24927d6fb383-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9wsvs\" (UID: \"031e6c9b-93f8-458b-a20d-24927d6fb383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"
Nov 03 10:01:17.547104 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:17.547040    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/031e6c9b-93f8-458b-a20d-24927d6fb383-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9wsvs\" (UID: \"031e6c9b-93f8-458b-a20d-24927d6fb383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"
Nov 03 10:01:17.547234 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:17.547214    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/031e6c9b-93f8-458b-a20d-24927d6fb383-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9wsvs\" (UID: \"031e6c9b-93f8-458b-a20d-24927d6fb383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"
Nov 03 10:01:17.548842 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:17.548822    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/031e6c9b-93f8-458b-a20d-24927d6fb383-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9wsvs\" (UID: \"031e6c9b-93f8-458b-a20d-24927d6fb383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"
Nov 03 10:01:17.548971 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:17.548938    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/031e6c9b-93f8-458b-a20d-24927d6fb383-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9wsvs\" (UID: \"031e6c9b-93f8-458b-a20d-24927d6fb383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"
Nov 03 10:01:17.605700 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:17.605677    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/031e6c9b-93f8-458b-a20d-24927d6fb383-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9wsvs\" (UID: \"031e6c9b-93f8-458b-a20d-24927d6fb383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"
Nov 03 10:01:17.606413 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:17.606393    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2jtg\" (UniqueName: \"kubernetes.io/projected/031e6c9b-93f8-458b-a20d-24927d6fb383-kube-api-access-b2jtg\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9wsvs\" (UID: \"031e6c9b-93f8-458b-a20d-24927d6fb383\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"
Nov 03 10:01:17.725902 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:17.725832    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"
Nov 03 10:01:18.152028 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:18.152002    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"]
Nov 03 10:01:18.153457 ip-10-0-139-73 kubenswrapper[4473]: W1103 10:01:18.153433    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod031e6c9b_93f8_458b_a20d_24927d6fb383.slice/crio-7838ab31903353a3f30a0930029ab6da6a2bb8a382c789ecb7d75e6ccb5b01a4 WatchSource:0}: Error finding container 7838ab31903353a3f30a0930029ab6da6a2bb8a382c789ecb7d75e6ccb5b01a4: Status 404 returned error can't find the container with id 7838ab31903353a3f30a0930029ab6da6a2bb8a382c789ecb7d75e6ccb5b01a4
Nov 03 10:01:18.155614 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:18.155581    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:01:18.155719 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:18.155648    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:01:18.155719 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:18.155683    4473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 03 10:01:18.295261 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:18.295225    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs" event={"ID":"031e6c9b-93f8-458b-a20d-24927d6fb383","Type":"ContainerStarted","Data":"67004f48df7494bae6905dac1007d5233f73b00c7808dad8610a54665aed4321"}
Nov 03 10:01:18.295429 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:18.295307    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs" event={"ID":"031e6c9b-93f8-458b-a20d-24927d6fb383","Type":"ContainerStarted","Data":"7838ab31903353a3f30a0930029ab6da6a2bb8a382c789ecb7d75e6ccb5b01a4"}
Nov 03 10:01:18.415756 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:18.415645    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs" podStartSLOduration=1.415627091 podStartE2EDuration="1.415627091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:01:18.413493296 +0000 UTC m=+4902.682556730" watchObservedRunningTime="2026-11-03 10:01:18.415627091 +0000 UTC m=+4902.684690527"
Nov 03 10:01:18.582763 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:18.582725    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:01:18.732026 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:18.729059    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"
Nov 03 10:01:19.732975 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:19.731158    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Nov 03 10:01:19.732975 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:19.731239    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs" podUID="031e6c9b-93f8-458b-a20d-24927d6fb383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Nov 03 10:01:20.730216 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:20.730181    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Nov 03 10:01:20.730519 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:20.730486    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs" podUID="031e6c9b-93f8-458b-a20d-24927d6fb383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Nov 03 10:01:21.734972 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:21.733343    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Nov 03 10:01:21.734972 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:21.733403    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs" podUID="031e6c9b-93f8-458b-a20d-24927d6fb383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Nov 03 10:01:22.732395 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:22.732357    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Nov 03 10:01:22.733527 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:22.733495    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs" podUID="031e6c9b-93f8-458b-a20d-24927d6fb383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Nov 03 10:01:23.734979 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:23.734379    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Nov 03 10:01:23.734979 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:23.734446    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs" podUID="031e6c9b-93f8-458b-a20d-24927d6fb383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Nov 03 10:01:24.726673 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:24.726631    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body=
Nov 03 10:01:24.726868 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:24.726687    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs" podUID="031e6c9b-93f8-458b-a20d-24927d6fb383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused"
Nov 03 10:01:26.726619 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:26.726569    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:01:26.727113 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:26.726640    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs" podUID="031e6c9b-93f8-458b-a20d-24927d6fb383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:01:27.768103 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:27.768053    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:01:27.768646 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:27.768108    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs" podUID="031e6c9b-93f8-458b-a20d-24927d6fb383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:01:28.768980 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:28.768910    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:01:28.769446 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:28.768997    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs" podUID="031e6c9b-93f8-458b-a20d-24927d6fb383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:01:29.813094 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:29.813058    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:01:29.813505 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:29.813113    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs" podUID="031e6c9b-93f8-458b-a20d-24927d6fb383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:01:30.857792 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:30.857068    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:01:30.857792 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:30.857132    4473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs" podUID="031e6c9b-93f8-458b-a20d-24927d6fb383" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:01:31.288741 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:31.288707    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"
Nov 03 10:01:31.289523 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:31.289499    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"
Nov 03 10:01:31.290303 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:31.290282    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"
Nov 03 10:01:39.350661 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:39.350625    4473 scope.go:122] "RemoveContainer" containerID="fb206098c0fead159851f1854855bdedc485622108ee717fbf416eea18e0e27d"
Nov 03 10:01:39.359047 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:39.359025    4473 scope.go:122] "RemoveContainer" containerID="523a0a5a38007e0eb7723f85a12de1e2cae431309429c213382f0c27ada5dac6"
Nov 03 10:01:40.052739 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:40.052694    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dcd4551-2fzzz"]
Nov 03 10:01:40.102870 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:40.102839    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dcd4551-2fzzz"]
Nov 03 10:01:41.868328 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:41.868298    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61529bbf-0a40-447b-b166-5bb4e6836cce" path="/var/lib/kubelet/pods/61529bbf-0a40-447b-b166-5bb4e6836cce/volumes"
Nov 03 10:01:46.936541 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:01:46.936504    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:01:50.052672 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:50.052631    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71076da4-bnqjf"]
Nov 03 10:01:50.103268 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:50.103239    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71076da4-bnqjf"]
Nov 03 10:01:51.868876 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:51.868847    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7c91a81-8f4f-4161-9189-01d948f1b55c" path="/var/lib/kubelet/pods/a7c91a81-8f4f-4161-9189-01d948f1b55c/volumes"
Nov 03 10:01:56.557486 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:01:56.557457    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:02:03.051164 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:02:03.051122    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e3864c4-p9tv6"]
Nov 03 10:02:03.101157 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:02:03.101129    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e3864c4-p9tv6"]
Nov 03 10:02:03.870746 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:02:03.870702    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8582e32-8742-478d-9cf8-67a8488ab67c" path="/var/lib/kubelet/pods/d8582e32-8742-478d-9cf8-67a8488ab67c/volumes"
Nov 03 10:02:10.477024 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:02:10.476977    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:02:16.936232 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:02:16.936188    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:02:39.380274 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:02:39.380246    4473 scope.go:122] "RemoveContainer" containerID="4a669c56d60c2e7d7a7ffd24c9d64f4e03d398d318286906facf227ecc8e3b4f"
Nov 03 10:02:46.936702 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:02:46.936604    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:02:52.244161 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:02:52.244122    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:03:16.935884 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:03:16.935849    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:03:26.750319 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:03:26.750280    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:03:38.139929 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:03:38.139891    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:03:39.399605 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:03:39.399573    4473 scope.go:122] "RemoveContainer" containerID="4a6d51bdbfaa8778e6ee00d216c5ef0056bc9e4653481a6a5bb8dd36676e0110"
Nov 03 10:03:39.407829 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:03:39.407805    4473 scope.go:122] "RemoveContainer" containerID="de44b78155fa522d5d3676f048b7124cb0506509c0dfedeb8bee7da7f0babb4d"
Nov 03 10:03:46.964231 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:03:46.964189    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:04:04.895961 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:04:04.895912    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:04:16.937416 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:04:16.937332    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:04:24.787728 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:04:24.787691    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:04:36.934350 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:04:36.934319    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 10:04:36.938153 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:04:36.938132    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 10:04:46.936582 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:04:46.936543    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:05:03.679009 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:05:03.678967    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:05:16.935964 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:05:16.935907    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:05:18.660648 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:05:18.660609    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4e36e81-gdmqv"]
Nov 03 10:05:18.664122 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:05:18.664106    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e36e81-gdmqv"
Nov 03 10:05:18.719911 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:05:18.719881    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 10:05:18.733996 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:05:18.733966    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t226\" (UniqueName: \"kubernetes.io/projected/93efdd5a-b563-4fc0-b4cf-f5ed61539486-kube-api-access-7t226\") pod \"maas-api-revoke-sub-b4e36e81-gdmqv\" (UID: \"93efdd5a-b563-4fc0-b4cf-f5ed61539486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e36e81-gdmqv"
Nov 03 10:05:18.734139 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:05:18.734017    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93efdd5a-b563-4fc0-b4cf-f5ed61539486-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4e36e81-gdmqv\" (UID: \"93efdd5a-b563-4fc0-b4cf-f5ed61539486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e36e81-gdmqv"
Nov 03 10:05:18.734139 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:05:18.734102    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93efdd5a-b563-4fc0-b4cf-f5ed61539486-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4e36e81-gdmqv\" (UID: \"93efdd5a-b563-4fc0-b4cf-f5ed61539486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e36e81-gdmqv"
Nov 03 10:05:18.770536 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:05:18.770506    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 10:05:18.772206 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:05:18.772184    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4e36e81-gdmqv"]
Nov 03 10:05:18.772377 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:05:18.772361    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 10:05:18.821408 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:05:18.821384    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\""
Nov 03 10:05:18.834972 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:05:18.834934    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t226\" (UniqueName: \"kubernetes.io/projected/93efdd5a-b563-4fc0-b4cf-f5ed61539486-kube-api-access-7t226\") pod \"maas-api-revoke-sub-b4e36e81-gdmqv\" (UID: \"93efdd5a-b563-4fc0-b4cf-f5ed61539486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e36e81-gdmqv"
Nov 03 10:05:18.835086 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:05:18.834992    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93efdd5a-b563-4fc0-b4cf-f5ed61539486-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4e36e81-gdmqv\" (UID: \"93efdd5a-b563-4fc0-b4cf-f5ed61539486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e36e81-gdmqv"
Nov 03 10:05:18.835086 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:05:18.835024    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93efdd5a-b563-4fc0-b4cf-f5ed61539486-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4e36e81-gdmqv\" (UID: \"93efdd5a-b563-4fc0-b4cf-f5ed61539486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e36e81-gdmqv"
Nov 03 10:05:18.835632 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:05:18.835612    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93efdd5a-b563-4fc0-b4cf-f5ed61539486-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4e36e81-gdmqv\" (UID: \"93efdd5a-b563-4fc0-b4cf-f5ed61539486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e36e81-gdmqv"
Nov 03 10:05:18.837595 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:05:18.837576    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93efdd5a-b563-4fc0-b4cf-f5ed61539486-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4e36e81-gdmqv\" (UID: \"93efdd5a-b563-4fc0-b4cf-f5ed61539486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e36e81-gdmqv"
Nov 03 10:05:18.894242 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:05:18.894218    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t226\" (UniqueName: \"kubernetes.io/projected/93efdd5a-b563-4fc0-b4cf-f5ed61539486-kube-api-access-7t226\") pod \"maas-api-revoke-sub-b4e36e81-gdmqv\" (UID: \"93efdd5a-b563-4fc0-b4cf-f5ed61539486\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e36e81-gdmqv"
Nov 03 10:05:18.977339 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:05:18.977267    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e36e81-gdmqv"
Nov 03 10:05:19.348385 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:05:19.348216    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4e36e81-gdmqv"]
Nov 03 10:05:19.350679 ip-10-0-139-73 kubenswrapper[4473]: W1103 10:05:19.350649    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93efdd5a_b563_4fc0_b4cf_f5ed61539486.slice/crio-ead9acd07e7d278886ef229f7dc4e19e3f7c0ec9c6b926f65030253706d6f802 WatchSource:0}: Error finding container ead9acd07e7d278886ef229f7dc4e19e3f7c0ec9c6b926f65030253706d6f802: Status 404 returned error can't find the container with id ead9acd07e7d278886ef229f7dc4e19e3f7c0ec9c6b926f65030253706d6f802
Nov 03 10:05:19.488245 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:05:19.488213    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e36e81-gdmqv" event={"ID":"93efdd5a-b563-4fc0-b4cf-f5ed61539486","Type":"ContainerStarted","Data":"ead9acd07e7d278886ef229f7dc4e19e3f7c0ec9c6b926f65030253706d6f802"}
Nov 03 10:05:20.494707 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:05:20.494624    4473 generic.go:356] "Generic (PLEG): container finished" podID="93efdd5a-b563-4fc0-b4cf-f5ed61539486" containerID="f68cb6f943aec04850e2c5c60fd5b74694be0281001e516c35344c3937166a31" exitCode=0
Nov 03 10:05:20.494707 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:05:20.494676    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e36e81-gdmqv" event={"ID":"93efdd5a-b563-4fc0-b4cf-f5ed61539486","Type":"ContainerDied","Data":"f68cb6f943aec04850e2c5c60fd5b74694be0281001e516c35344c3937166a31"}
Nov 03 10:05:21.627341 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:05:21.627319    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e36e81-gdmqv"
Nov 03 10:05:21.660206 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:05:21.660177    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93efdd5a-b563-4fc0-b4cf-f5ed61539486-kube-api-access-7t226\" (UniqueName: \"kubernetes.io/projected/93efdd5a-b563-4fc0-b4cf-f5ed61539486-kube-api-access-7t226\") pod \"93efdd5a-b563-4fc0-b4cf-f5ed61539486\" (UID: \"93efdd5a-b563-4fc0-b4cf-f5ed61539486\") "
Nov 03 10:05:21.660349 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:05:21.660218    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93efdd5a-b563-4fc0-b4cf-f5ed61539486-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93efdd5a-b563-4fc0-b4cf-f5ed61539486-maas-api-xks-ca\") pod \"93efdd5a-b563-4fc0-b4cf-f5ed61539486\" (UID: \"93efdd5a-b563-4fc0-b4cf-f5ed61539486\") "
Nov 03 10:05:21.660349 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:05:21.660288    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93efdd5a-b563-4fc0-b4cf-f5ed61539486-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93efdd5a-b563-4fc0-b4cf-f5ed61539486-maas-api-openshift-ca\") pod \"93efdd5a-b563-4fc0-b4cf-f5ed61539486\" (UID: \"93efdd5a-b563-4fc0-b4cf-f5ed61539486\") "
Nov 03 10:05:21.660635 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:05:21.660614    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93efdd5a-b563-4fc0-b4cf-f5ed61539486-maas-api-openshift-ca" pod "93efdd5a-b563-4fc0-b4cf-f5ed61539486" (UID: "93efdd5a-b563-4fc0-b4cf-f5ed61539486"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:05:21.662311 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:05:21.662290    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93efdd5a-b563-4fc0-b4cf-f5ed61539486-kube-api-access-7t226" pod "93efdd5a-b563-4fc0-b4cf-f5ed61539486" (UID: "93efdd5a-b563-4fc0-b4cf-f5ed61539486"). InnerVolumeSpecName "kube-api-access-7t226". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:05:21.662428 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:05:21.662382    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93efdd5a-b563-4fc0-b4cf-f5ed61539486-maas-api-xks-ca" pod "93efdd5a-b563-4fc0-b4cf-f5ed61539486" (UID: "93efdd5a-b563-4fc0-b4cf-f5ed61539486"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:05:21.761631 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:05:21.761573    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t226\" (UniqueName: \"kubernetes.io/projected/93efdd5a-b563-4fc0-b4cf-f5ed61539486-kube-api-access-7t226\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 10:05:21.761631 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:05:21.761596    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93efdd5a-b563-4fc0-b4cf-f5ed61539486-maas-api-xks-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 10:05:21.761631 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:05:21.761605    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93efdd5a-b563-4fc0-b4cf-f5ed61539486-maas-api-openshift-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 10:05:22.504540 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:05:22.504505    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e36e81-gdmqv"
Nov 03 10:05:22.504540 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:05:22.504531    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e36e81-gdmqv" event={"ID":"93efdd5a-b563-4fc0-b4cf-f5ed61539486","Type":"ContainerDied","Data":"ead9acd07e7d278886ef229f7dc4e19e3f7c0ec9c6b926f65030253706d6f802"}
Nov 03 10:05:22.504767 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:05:22.504578    4473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ead9acd07e7d278886ef229f7dc4e19e3f7c0ec9c6b926f65030253706d6f802"
Nov 03 10:05:31.781420 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:05:31.781389    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:05:42.181430 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:05:42.181395    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:05:46.936306 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:05:46.936233    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:06:16.936812 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:06:16.936774    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:06:18.382795 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:06:18.382741    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:06:46.939075 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:06:46.939038    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:06:48.710812 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:06:48.710777    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:07:08.883284 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:07:08.883250    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:07:16.936074 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:07:16.935988    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:07:40.782873 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:07:40.782841    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:07:46.936959 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:07:46.936910    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:08:10.144542 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:08:10.144506    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:08:16.935598 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:08:16.935559    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:08:25.868524 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:08:25.868488    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:08:42.792234 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:08:42.792200    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:08:46.936430 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:08:46.936355    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:08:55.277555 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:08:55.277512    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:09:05.485063 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:09:05.485018    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:09:16.936461 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:09:16.936425    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:09:19.979587 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:09:19.979556    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:09:36.969755 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:09:36.969722    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 10:09:36.971872 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:09:36.971849    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 10:09:46.936448 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:09:46.936412    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:10:16.936851 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:10:16.936777    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:10:21.496843 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:10:21.496802    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:10:22.054639 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:10:22.054606    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4e36e81-gdmqv"]
Nov 03 10:10:22.104242 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:10:22.104207    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4e36e81-gdmqv"]
Nov 03 10:10:23.885972 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:10:23.885000    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93efdd5a-b563-4fc0-b4cf-f5ed61539486" path="/var/lib/kubelet/pods/93efdd5a-b563-4fc0-b4cf-f5ed61539486/volumes"
Nov 03 10:10:46.936221 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:10:46.936182    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:11:16.936855 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:11:16.936808    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:11:39.479788 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:11:39.479751    4473 scope.go:122] "RemoveContainer" containerID="f68cb6f943aec04850e2c5c60fd5b74694be0281001e516c35344c3937166a31"
Nov 03 10:11:46.937029 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:11:46.936935    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:12:16.935855 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:12:16.935815    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:12:19.983776 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:12:19.983733    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:12:30.743299 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:12:30.743259    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:12:46.936695 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:12:46.936650    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:12:49.082833 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:12:49.082800    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:13:00.188279 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:13:00.188247    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:13:14.588547 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:13:14.588512    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:13:16.935547 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:13:16.935460    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:13:41.376960 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:13:41.376911    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:13:46.934604 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:13:46.934567    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:14:16.936215 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:14:16.936181    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:14:22.196273 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:14:22.196242    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:14:32.891325 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:14:32.891292    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:14:36.997208 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:14:36.997174    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 10:14:37.000039 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:14:37.000012    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 10:14:46.170134 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:14:46.170100    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:14:46.935490 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:14:46.935414    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:15:00.156164 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:00.156128    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895015-sv8tt"]
Nov 03 10:15:00.156681 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:00.156488    4473 cpu_manager.go:414] "containerMap: removing container" podUID="93efdd5a-b563-4fc0-b4cf-f5ed61539486" containerName="revoke-keys"
Nov 03 10:15:00.156681 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:00.156498    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93efdd5a-b563-4fc0-b4cf-f5ed61539486" containerName="revoke-keys"
Nov 03 10:15:00.156681 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:00.156574    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="93efdd5a-b563-4fc0-b4cf-f5ed61539486" containerName="revoke-keys"
Nov 03 10:15:00.159593 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:00.159578    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895015-sv8tt"
Nov 03 10:15:00.216690 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:00.216647    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895015-sv8tt"]
Nov 03 10:15:00.245676 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:00.245639    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f9hc\" (UniqueName: \"kubernetes.io/projected/f1f33b3d-b144-463c-abe9-319e2fe0568f-kube-api-access-4f9hc\") pod \"maas-api-key-cleanup-29895015-sv8tt\" (UID: \"f1f33b3d-b144-463c-abe9-319e2fe0568f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895015-sv8tt"
Nov 03 10:15:00.245870 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:00.245722    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1f33b3d-b144-463c-abe9-319e2fe0568f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895015-sv8tt\" (UID: \"f1f33b3d-b144-463c-abe9-319e2fe0568f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895015-sv8tt"
Nov 03 10:15:00.264730 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:00.264696    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 10:15:00.264730 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:00.264729    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 10:15:00.264925 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:00.264823    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnmdh\""
Nov 03 10:15:00.347033 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:00.346998    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1f33b3d-b144-463c-abe9-319e2fe0568f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895015-sv8tt\" (UID: \"f1f33b3d-b144-463c-abe9-319e2fe0568f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895015-sv8tt"
Nov 03 10:15:00.347229 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:00.347069    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f9hc\" (UniqueName: \"kubernetes.io/projected/f1f33b3d-b144-463c-abe9-319e2fe0568f-kube-api-access-4f9hc\") pod \"maas-api-key-cleanup-29895015-sv8tt\" (UID: \"f1f33b3d-b144-463c-abe9-319e2fe0568f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895015-sv8tt"
Nov 03 10:15:00.347741 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:00.347721    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1f33b3d-b144-463c-abe9-319e2fe0568f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895015-sv8tt\" (UID: \"f1f33b3d-b144-463c-abe9-319e2fe0568f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895015-sv8tt"
Nov 03 10:15:00.408302 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:00.408230    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f9hc\" (UniqueName: \"kubernetes.io/projected/f1f33b3d-b144-463c-abe9-319e2fe0568f-kube-api-access-4f9hc\") pod \"maas-api-key-cleanup-29895015-sv8tt\" (UID: \"f1f33b3d-b144-463c-abe9-319e2fe0568f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895015-sv8tt"
Nov 03 10:15:00.471921 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:00.471885    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895015-sv8tt"
Nov 03 10:15:00.845369 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:00.845343    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895015-sv8tt"]
Nov 03 10:15:00.847716 ip-10-0-139-73 kubenswrapper[4473]: W1103 10:15:00.847688    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1f33b3d_b144_463c_abe9_319e2fe0568f.slice/crio-f5425b6935ea859814cd21381095bb2b425a90b1837e2ac627eb4c446f6ad6b4 WatchSource:0}: Error finding container f5425b6935ea859814cd21381095bb2b425a90b1837e2ac627eb4c446f6ad6b4: Status 404 returned error can't find the container with id f5425b6935ea859814cd21381095bb2b425a90b1837e2ac627eb4c446f6ad6b4
Nov 03 10:15:01.279086 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:01.279039    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895015-sv8tt" event={"ID":"f1f33b3d-b144-463c-abe9-319e2fe0568f","Type":"ContainerStarted","Data":"55f886672dc21a1f2a2fb07bde660312b1561bbf9f8d88e0ca3b577e7a597b2e"}
Nov 03 10:15:01.279555 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:01.279119    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895015-sv8tt" event={"ID":"f1f33b3d-b144-463c-abe9-319e2fe0568f","Type":"ContainerStarted","Data":"f5425b6935ea859814cd21381095bb2b425a90b1837e2ac627eb4c446f6ad6b4"}
Nov 03 10:15:01.394638 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:01.394543    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895015-sv8tt" podStartSLOduration=1.394532203 podStartE2EDuration="1.394532203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:15:01.39373541 +0000 UTC m=+5725.662798864" watchObservedRunningTime="2026-11-03 10:15:01.394532203 +0000 UTC m=+5725.663595637"
Nov 03 10:15:02.285262 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:02.285221    4473 generic.go:356] "Generic (PLEG): container finished" podID="f1f33b3d-b144-463c-abe9-319e2fe0568f" containerID="55f886672dc21a1f2a2fb07bde660312b1561bbf9f8d88e0ca3b577e7a597b2e" exitCode=60
Nov 03 10:15:02.285760 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:02.285452    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895015-sv8tt" event={"ID":"f1f33b3d-b144-463c-abe9-319e2fe0568f","Type":"ContainerDied","Data":"55f886672dc21a1f2a2fb07bde660312b1561bbf9f8d88e0ca3b577e7a597b2e"}
Nov 03 10:15:02.285760 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:02.285550    4473 scope.go:122] "RemoveContainer" containerID="55f886672dc21a1f2a2fb07bde660312b1561bbf9f8d88e0ca3b577e7a597b2e"
Nov 03 10:15:03.291623 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:03.291528    4473 generic.go:356] "Generic (PLEG): container finished" podID="f1f33b3d-b144-463c-abe9-319e2fe0568f" containerID="aac53454c81243ea666b8ce112d778161352675376f8422e93d2c12b729e7fb8" exitCode=60
Nov 03 10:15:03.291623 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:03.291582    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895015-sv8tt" event={"ID":"f1f33b3d-b144-463c-abe9-319e2fe0568f","Type":"ContainerDied","Data":"aac53454c81243ea666b8ce112d778161352675376f8422e93d2c12b729e7fb8"}
Nov 03 10:15:03.292267 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:03.291629    4473 scope.go:122] "RemoveContainer" containerID="55f886672dc21a1f2a2fb07bde660312b1561bbf9f8d88e0ca3b577e7a597b2e"
Nov 03 10:15:03.292267 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:03.291898    4473 scope.go:122] "RemoveContainer" containerID="aac53454c81243ea666b8ce112d778161352675376f8422e93d2c12b729e7fb8"
Nov 03 10:15:03.292267 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:15:03.292167    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895015-sv8tt_odh-ai-gateway-infra(f1f33b3d-b144-463c-abe9-319e2fe0568f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895015-sv8tt" podUID="f1f33b3d-b144-463c-abe9-319e2fe0568f"
Nov 03 10:15:04.297555 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:04.297527    4473 scope.go:122] "RemoveContainer" containerID="aac53454c81243ea666b8ce112d778161352675376f8422e93d2c12b729e7fb8"
Nov 03 10:15:04.297970 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:15:04.297753    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895015-sv8tt_odh-ai-gateway-infra(f1f33b3d-b144-463c-abe9-319e2fe0568f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895015-sv8tt" podUID="f1f33b3d-b144-463c-abe9-319e2fe0568f"
Nov 03 10:15:10.783215 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:10.783181    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:15:15.868682 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:15.868652    4473 scope.go:122] "RemoveContainer" containerID="aac53454c81243ea666b8ce112d778161352675376f8422e93d2c12b729e7fb8"
Nov 03 10:15:16.360525 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:16.360482    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895015-sv8tt" event={"ID":"f1f33b3d-b144-463c-abe9-319e2fe0568f","Type":"ContainerStarted","Data":"bb5916ec5d2f356558f42ae61d39ad06f40c22f6aa291e82a011103754a81ff2"}
Nov 03 10:15:17.005013 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:15:17.004906    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:15:17.398837 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:17.398801    4473 generic.go:356] "Generic (PLEG): container finished" podID="f1f33b3d-b144-463c-abe9-319e2fe0568f" containerID="bb5916ec5d2f356558f42ae61d39ad06f40c22f6aa291e82a011103754a81ff2" exitCode=60
Nov 03 10:15:17.399086 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:17.398881    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895015-sv8tt" event={"ID":"f1f33b3d-b144-463c-abe9-319e2fe0568f","Type":"ContainerDied","Data":"bb5916ec5d2f356558f42ae61d39ad06f40c22f6aa291e82a011103754a81ff2"}
Nov 03 10:15:17.399086 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:17.398982    4473 scope.go:122] "RemoveContainer" containerID="aac53454c81243ea666b8ce112d778161352675376f8422e93d2c12b729e7fb8"
Nov 03 10:15:17.399307 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:17.399287    4473 scope.go:122] "RemoveContainer" containerID="bb5916ec5d2f356558f42ae61d39ad06f40c22f6aa291e82a011103754a81ff2"
Nov 03 10:15:17.399600 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:15:17.399572    4473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895015-sv8tt_odh-ai-gateway-infra(f1f33b3d-b144-463c-abe9-319e2fe0568f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895015-sv8tt" podUID="f1f33b3d-b144-463c-abe9-319e2fe0568f"
Nov 03 10:15:17.509575 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:17.509535    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895015-sv8tt"]
Nov 03 10:15:18.699013 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:18.698986    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895015-sv8tt"
Nov 03 10:15:18.797559 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:18.797529    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1f33b3d-b144-463c-abe9-319e2fe0568f-kube-api-access-4f9hc\" (UniqueName: \"kubernetes.io/projected/f1f33b3d-b144-463c-abe9-319e2fe0568f-kube-api-access-4f9hc\") pod \"f1f33b3d-b144-463c-abe9-319e2fe0568f\" (UID: \"f1f33b3d-b144-463c-abe9-319e2fe0568f\") "
Nov 03 10:15:18.797730 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:18.797569    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1f33b3d-b144-463c-abe9-319e2fe0568f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1f33b3d-b144-463c-abe9-319e2fe0568f-maas-api-service-ca\") pod \"f1f33b3d-b144-463c-abe9-319e2fe0568f\" (UID: \"f1f33b3d-b144-463c-abe9-319e2fe0568f\") "
Nov 03 10:15:18.798058 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:18.797940    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1f33b3d-b144-463c-abe9-319e2fe0568f-maas-api-service-ca" pod "f1f33b3d-b144-463c-abe9-319e2fe0568f" (UID: "f1f33b3d-b144-463c-abe9-319e2fe0568f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:15:18.799832 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:18.799804    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1f33b3d-b144-463c-abe9-319e2fe0568f-kube-api-access-4f9hc" pod "f1f33b3d-b144-463c-abe9-319e2fe0568f" (UID: "f1f33b3d-b144-463c-abe9-319e2fe0568f"). InnerVolumeSpecName "kube-api-access-4f9hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:15:18.868167 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:18.868134    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:15:18.899178 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:18.899153    4473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1f33b3d-b144-463c-abe9-319e2fe0568f-maas-api-service-ca\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 10:15:18.899178 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:18.899177    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f9hc\" (UniqueName: \"kubernetes.io/projected/f1f33b3d-b144-463c-abe9-319e2fe0568f-kube-api-access-4f9hc\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 10:15:19.412332 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:19.412302    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895015-sv8tt" event={"ID":"f1f33b3d-b144-463c-abe9-319e2fe0568f","Type":"ContainerDied","Data":"f5425b6935ea859814cd21381095bb2b425a90b1837e2ac627eb4c446f6ad6b4"}
Nov 03 10:15:19.412332 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:19.412324    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895015-sv8tt"
Nov 03 10:15:19.412566 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:19.412362    4473 scope.go:122] "RemoveContainer" containerID="bb5916ec5d2f356558f42ae61d39ad06f40c22f6aa291e82a011103754a81ff2"
Nov 03 10:15:19.580054 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:19.580023    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895015-sv8tt"]
Nov 03 10:15:19.586146 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:19.586115    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895015-sv8tt"]
Nov 03 10:15:19.869217 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:19.869185    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1f33b3d-b144-463c-abe9-319e2fe0568f" path="/var/lib/kubelet/pods/f1f33b3d-b144-463c-abe9-319e2fe0568f/volumes"
Nov 03 10:15:27.682327 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:27.682295    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:15:38.072243 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:15:38.072200    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:15:46.936610 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:15:46.936501    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:16:04.658452 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:16:04.658418    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:16:15.540919 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:16:15.540883    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:16:16.935608 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:16:16.935523    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:16:33.553957 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:16:33.553917    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:16:46.936410 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:16:46.936374    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:17:03.269025 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:17:03.268990    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:17:16.937047 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:17:16.937012    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:17:24.980987 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:17:24.980936    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:17:35.666152 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:17:35.666115    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:17:46.939521 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:17:46.939439    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:18:16.498609 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:16.498574    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:18:16.936862 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:18:16.936830    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:18:29.868429 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:29.868402    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-65cb27/simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm"]
Nov 03 10:18:29.868816 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:29.868734    4473 cpu_manager.go:414] "containerMap: removing container" podUID="f1f33b3d-b144-463c-abe9-319e2fe0568f" containerName="cleanup"
Nov 03 10:18:29.868816 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:29.868746    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1f33b3d-b144-463c-abe9-319e2fe0568f" containerName="cleanup"
Nov 03 10:18:29.868816 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:29.868766    4473 cpu_manager.go:414] "containerMap: removing container" podUID="f1f33b3d-b144-463c-abe9-319e2fe0568f" containerName="cleanup"
Nov 03 10:18:29.868816 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:29.868771    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1f33b3d-b144-463c-abe9-319e2fe0568f" containerName="cleanup"
Nov 03 10:18:29.868816 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:29.868782    4473 cpu_manager.go:414] "containerMap: removing container" podUID="f1f33b3d-b144-463c-abe9-319e2fe0568f" containerName="cleanup"
Nov 03 10:18:29.868816 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:29.868787    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1f33b3d-b144-463c-abe9-319e2fe0568f" containerName="cleanup"
Nov 03 10:18:29.869049 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:29.868849    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1f33b3d-b144-463c-abe9-319e2fe0568f" containerName="cleanup"
Nov 03 10:18:29.869049 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:29.868858    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1f33b3d-b144-463c-abe9-319e2fe0568f" containerName="cleanup"
Nov 03 10:18:29.871714 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:29.871699    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-65cb27/simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm"
Nov 03 10:18:29.956350 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:29.956315    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-65cb27/simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm"]
Nov 03 10:18:29.970092 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:29.970065    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-dshm\") pod \"simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm\" (UID: \"f7dc7ea7-0abf-4f06-89dc-e79520dcd889\") " pod="e2e-models-e2e-worker-single-main-65cb27/simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm"
Nov 03 10:18:29.970248 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:29.970098    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-tls-certs\") pod \"simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm\" (UID: \"f7dc7ea7-0abf-4f06-89dc-e79520dcd889\") " pod="e2e-models-e2e-worker-single-main-65cb27/simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm"
Nov 03 10:18:29.970248 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:29.970186    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-home\") pod \"simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm\" (UID: \"f7dc7ea7-0abf-4f06-89dc-e79520dcd889\") " pod="e2e-models-e2e-worker-single-main-65cb27/simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm"
Nov 03 10:18:29.970379 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:29.970292    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-model-cache\") pod \"simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm\" (UID: \"f7dc7ea7-0abf-4f06-89dc-e79520dcd889\") " pod="e2e-models-e2e-worker-single-main-65cb27/simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm"
Nov 03 10:18:29.970379 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:29.970370    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpxph\" (UniqueName: \"kubernetes.io/projected/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-kube-api-access-tpxph\") pod \"simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm\" (UID: \"f7dc7ea7-0abf-4f06-89dc-e79520dcd889\") " pod="e2e-models-e2e-worker-single-main-65cb27/simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm"
Nov 03 10:18:29.970495 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:29.970401    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-tmp-dir\") pod \"simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm\" (UID: \"f7dc7ea7-0abf-4f06-89dc-e79520dcd889\") " pod="e2e-models-e2e-worker-single-main-65cb27/simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm"
Nov 03 10:18:29.977735 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:29.977716    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-65cb27\"/\"kube-root-ca.crt\""
Nov 03 10:18:29.977987 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:29.977970    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-65cb27\"/\"openshift-service-ca.crt\""
Nov 03 10:18:29.978038 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:29.978000    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-65cb27\"/\"simulator-single-main-65cb27-kserve-self-signed-certs\""
Nov 03 10:18:29.978079 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:29.978057    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-65cb27\"/\"default-dockercfg-s8k29\""
Nov 03 10:18:30.071093 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:30.071058    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-home\") pod \"simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm\" (UID: \"f7dc7ea7-0abf-4f06-89dc-e79520dcd889\") " pod="e2e-models-e2e-worker-single-main-65cb27/simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm"
Nov 03 10:18:30.071299 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:30.071100    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-model-cache\") pod \"simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm\" (UID: \"f7dc7ea7-0abf-4f06-89dc-e79520dcd889\") " pod="e2e-models-e2e-worker-single-main-65cb27/simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm"
Nov 03 10:18:30.071299 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:30.071148    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpxph\" (UniqueName: \"kubernetes.io/projected/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-kube-api-access-tpxph\") pod \"simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm\" (UID: \"f7dc7ea7-0abf-4f06-89dc-e79520dcd889\") " pod="e2e-models-e2e-worker-single-main-65cb27/simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm"
Nov 03 10:18:30.071299 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:30.071170    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-tmp-dir\") pod \"simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm\" (UID: \"f7dc7ea7-0abf-4f06-89dc-e79520dcd889\") " pod="e2e-models-e2e-worker-single-main-65cb27/simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm"
Nov 03 10:18:30.071299 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:30.071219    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-dshm\") pod \"simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm\" (UID: \"f7dc7ea7-0abf-4f06-89dc-e79520dcd889\") " pod="e2e-models-e2e-worker-single-main-65cb27/simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm"
Nov 03 10:18:30.071299 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:30.071251    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-tls-certs\") pod \"simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm\" (UID: \"f7dc7ea7-0abf-4f06-89dc-e79520dcd889\") " pod="e2e-models-e2e-worker-single-main-65cb27/simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm"
Nov 03 10:18:30.071556 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:30.071532    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-home\") pod \"simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm\" (UID: \"f7dc7ea7-0abf-4f06-89dc-e79520dcd889\") " pod="e2e-models-e2e-worker-single-main-65cb27/simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm"
Nov 03 10:18:30.071603 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:30.071577    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-model-cache\") pod \"simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm\" (UID: \"f7dc7ea7-0abf-4f06-89dc-e79520dcd889\") " pod="e2e-models-e2e-worker-single-main-65cb27/simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm"
Nov 03 10:18:30.071639 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:30.071627    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-tmp-dir\") pod \"simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm\" (UID: \"f7dc7ea7-0abf-4f06-89dc-e79520dcd889\") " pod="e2e-models-e2e-worker-single-main-65cb27/simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm"
Nov 03 10:18:30.073612 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:30.073585    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-dshm\") pod \"simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm\" (UID: \"f7dc7ea7-0abf-4f06-89dc-e79520dcd889\") " pod="e2e-models-e2e-worker-single-main-65cb27/simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm"
Nov 03 10:18:30.074039 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:30.074007    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-tls-certs\") pod \"simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm\" (UID: \"f7dc7ea7-0abf-4f06-89dc-e79520dcd889\") " pod="e2e-models-e2e-worker-single-main-65cb27/simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm"
Nov 03 10:18:30.130979 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:30.130932    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpxph\" (UniqueName: \"kubernetes.io/projected/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-kube-api-access-tpxph\") pod \"simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm\" (UID: \"f7dc7ea7-0abf-4f06-89dc-e79520dcd889\") " pod="e2e-models-e2e-worker-single-main-65cb27/simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm"
Nov 03 10:18:30.183213 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:30.183184    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-65cb27/simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm"
Nov 03 10:18:30.565132 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:30.565106    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-65cb27/simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm"]
Nov 03 10:18:30.567194 ip-10-0-139-73 kubenswrapper[4473]: W1103 10:18:30.567166    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7dc7ea7_0abf_4f06_89dc_e79520dcd889.slice/crio-54befe2b3f1a900d52bae2d3bfaaadd30468826e5bd60cfc663d510c351f2370 WatchSource:0}: Error finding container 54befe2b3f1a900d52bae2d3bfaaadd30468826e5bd60cfc663d510c351f2370: Status 404 returned error can't find the container with id 54befe2b3f1a900d52bae2d3bfaaadd30468826e5bd60cfc663d510c351f2370
Nov 03 10:18:31.358137 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:31.358104    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-65cb27/simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm" event={"ID":"f7dc7ea7-0abf-4f06-89dc-e79520dcd889","Type":"ContainerStarted","Data":"44606565b359be3b99e141e0d2fb188a771f4f404215a2fe1807a6d3bd263e08"}
Nov 03 10:18:31.358137 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:31.358149    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-65cb27/simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm" event={"ID":"f7dc7ea7-0abf-4f06-89dc-e79520dcd889","Type":"ContainerStarted","Data":"54befe2b3f1a900d52bae2d3bfaaadd30468826e5bd60cfc663d510c351f2370"}
Nov 03 10:18:31.476078 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:31.476032    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-65cb27/simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm" podStartSLOduration=2.476019347 podStartE2EDuration="2.476019347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:18:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:18:31.474805447 +0000 UTC m=+5935.743868926" watchObservedRunningTime="2026-11-03 10:18:31.476019347 +0000 UTC m=+5935.745082806"
Nov 03 10:18:32.183959 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:32.183922    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-65cb27/simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm"
Nov 03 10:18:34.675813 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:34.675778    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:18:40.183856 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:40.183819    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-65cb27/simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm"
Nov 03 10:18:40.196578 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:40.196555    4473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-65cb27/simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm"
Nov 03 10:18:40.409225 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:18:40.409197    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-65cb27/simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm"
Nov 03 10:18:46.936897 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:18:46.936863    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:19:01.081883 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:19:01.081848    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:19:16.936524 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:19:16.936445    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:19:30.558154 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:19:30.558113    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:19:32.293122 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:19:32.293088    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-65cb27/simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm"]
Nov 03 10:19:37.032934 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:19:37.032906    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 10:19:37.037025 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:19:37.037002    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 10:19:42.392314 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:19:42.392278    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:19:46.936488 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:19:46.936453    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:19:47.313684 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:19:47.313596    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-65cb27/simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm" podUID="f7dc7ea7-0abf-4f06-89dc-e79520dcd889" containerName="main" containerID="cri-o://44606565b359be3b99e141e0d2fb188a771f4f404215a2fe1807a6d3bd263e08" gracePeriod=45
Nov 03 10:19:47.399483 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:19:47.399441    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-65cb27/simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm" podUID="f7dc7ea7-0abf-4f06-89dc-e79520dcd889" containerName="main" probeResult="failure" output="Get \"https://10.133.1.0:8000/ready\": dial tcp 10.133.1.0:8000: connect: connection refused"
Nov 03 10:19:47.563799 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:19:47.563738    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-65cb27/simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm"
Nov 03 10:19:47.590028 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:19:47.590003    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-home\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-home\") pod \"f7dc7ea7-0abf-4f06-89dc-e79520dcd889\" (UID: \"f7dc7ea7-0abf-4f06-89dc-e79520dcd889\") "
Nov 03 10:19:47.590196 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:19:47.590035    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-tmp-dir\") pod \"f7dc7ea7-0abf-4f06-89dc-e79520dcd889\" (UID: \"f7dc7ea7-0abf-4f06-89dc-e79520dcd889\") "
Nov 03 10:19:47.590196 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:19:47.590121    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-model-cache\") pod \"f7dc7ea7-0abf-4f06-89dc-e79520dcd889\" (UID: \"f7dc7ea7-0abf-4f06-89dc-e79520dcd889\") "
Nov 03 10:19:47.590325 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:19:47.590200    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-kube-api-access-tpxph\" (UniqueName: \"kubernetes.io/projected/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-kube-api-access-tpxph\") pod \"f7dc7ea7-0abf-4f06-89dc-e79520dcd889\" (UID: \"f7dc7ea7-0abf-4f06-89dc-e79520dcd889\") "
Nov 03 10:19:47.590325 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:19:47.590234    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-tls-certs\" (UniqueName: \"kubernetes.io/secret/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-tls-certs\") pod \"f7dc7ea7-0abf-4f06-89dc-e79520dcd889\" (UID: \"f7dc7ea7-0abf-4f06-89dc-e79520dcd889\") "
Nov 03 10:19:47.590325 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:19:47.590269    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-dshm\") pod \"f7dc7ea7-0abf-4f06-89dc-e79520dcd889\" (UID: \"f7dc7ea7-0abf-4f06-89dc-e79520dcd889\") "
Nov 03 10:19:47.590325 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:19:47.590278    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-home" pod "f7dc7ea7-0abf-4f06-89dc-e79520dcd889" (UID: "f7dc7ea7-0abf-4f06-89dc-e79520dcd889"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:19:47.590509 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:19:47.590356    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-tmp-dir" pod "f7dc7ea7-0abf-4f06-89dc-e79520dcd889" (UID: "f7dc7ea7-0abf-4f06-89dc-e79520dcd889"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:19:47.590509 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:19:47.590434    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-model-cache" pod "f7dc7ea7-0abf-4f06-89dc-e79520dcd889" (UID: "f7dc7ea7-0abf-4f06-89dc-e79520dcd889"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:19:47.590597 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:19:47.590579    4473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-home\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 10:19:47.590642 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:19:47.590604    4473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-tmp-dir\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 10:19:47.590642 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:19:47.590619    4473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-model-cache\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 10:19:47.592598 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:19:47.592574    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-dshm" pod "f7dc7ea7-0abf-4f06-89dc-e79520dcd889" (UID: "f7dc7ea7-0abf-4f06-89dc-e79520dcd889"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:19:47.592709 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:19:47.592647    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-tls-certs" pod "f7dc7ea7-0abf-4f06-89dc-e79520dcd889" (UID: "f7dc7ea7-0abf-4f06-89dc-e79520dcd889"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 10:19:47.592709 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:19:47.592694    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-kube-api-access-tpxph" pod "f7dc7ea7-0abf-4f06-89dc-e79520dcd889" (UID: "f7dc7ea7-0abf-4f06-89dc-e79520dcd889"). InnerVolumeSpecName "kube-api-access-tpxph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:19:47.691813 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:19:47.691773    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpxph\" (UniqueName: \"kubernetes.io/projected/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-kube-api-access-tpxph\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 10:19:47.691813 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:19:47.691808    4473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-tls-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 10:19:47.692035 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:19:47.691822    4473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7dc7ea7-0abf-4f06-89dc-e79520dcd889-dshm\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 10:19:47.695034 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:19:47.695007    4473 generic.go:356] "Generic (PLEG): container finished" podID="f7dc7ea7-0abf-4f06-89dc-e79520dcd889" containerID="44606565b359be3b99e141e0d2fb188a771f4f404215a2fe1807a6d3bd263e08" exitCode=0
Nov 03 10:19:47.695173 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:19:47.695052    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-65cb27/simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm"
Nov 03 10:19:47.695173 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:19:47.695089    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-65cb27/simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm" event={"ID":"f7dc7ea7-0abf-4f06-89dc-e79520dcd889","Type":"ContainerDied","Data":"44606565b359be3b99e141e0d2fb188a771f4f404215a2fe1807a6d3bd263e08"}
Nov 03 10:19:47.695173 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:19:47.695128    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-65cb27/simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm" event={"ID":"f7dc7ea7-0abf-4f06-89dc-e79520dcd889","Type":"ContainerDied","Data":"54befe2b3f1a900d52bae2d3bfaaadd30468826e5bd60cfc663d510c351f2370"}
Nov 03 10:19:47.695173 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:19:47.695145    4473 scope.go:122] "RemoveContainer" containerID="44606565b359be3b99e141e0d2fb188a771f4f404215a2fe1807a6d3bd263e08"
Nov 03 10:19:47.704762 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:19:47.704741    4473 scope.go:122] "RemoveContainer" containerID="44606565b359be3b99e141e0d2fb188a771f4f404215a2fe1807a6d3bd263e08"
Nov 03 10:19:47.705056 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:19:47.705036    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44606565b359be3b99e141e0d2fb188a771f4f404215a2fe1807a6d3bd263e08\": container with ID starting with 44606565b359be3b99e141e0d2fb188a771f4f404215a2fe1807a6d3bd263e08 not found: ID does not exist" containerID="44606565b359be3b99e141e0d2fb188a771f4f404215a2fe1807a6d3bd263e08"
Nov 03 10:19:47.705122 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:19:47.705066    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44606565b359be3b99e141e0d2fb188a771f4f404215a2fe1807a6d3bd263e08"} err="failed to get container status \"44606565b359be3b99e141e0d2fb188a771f4f404215a2fe1807a6d3bd263e08\": rpc error: code = NotFound desc = could not find container \"44606565b359be3b99e141e0d2fb188a771f4f404215a2fe1807a6d3bd263e08\": container with ID starting with 44606565b359be3b99e141e0d2fb188a771f4f404215a2fe1807a6d3bd263e08 not found: ID does not exist"
Nov 03 10:19:47.869337 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:19:47.869308    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-65cb27/simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm"]
Nov 03 10:19:47.869496 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:19:47.869351    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-65cb27/simulator-single-main-65cb27-kserve-d6f8dd797-f2hgm"]
Nov 03 10:19:49.868623 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:19:49.868588    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7dc7ea7-0abf-4f06-89dc-e79520dcd889" path="/var/lib/kubelet/pods/f7dc7ea7-0abf-4f06-89dc-e79520dcd889/volumes"
Nov 03 10:20:16.937322 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:20:16.937277    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:20:46.936525 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:20:46.936441    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:21:16.936357 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:21:16.936320    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:21:42.456651 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:21:42.456618    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:21:46.936188 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:21:46.936145    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:21:59.982084 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:21:59.982049    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:22:09.662458 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:22:09.662421    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:22:16.935801 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:22:16.935719    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:22:18.268798 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:22:18.268765    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:22:37.879555 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:22:37.879522    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:22:46.935989 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:22:46.935917    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:23:13.490642 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:23:13.490602    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:23:16.936610 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:23:16.936561    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:23:26.072675 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:23:26.072640    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:23:46.936453 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:23:46.936374    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:23:48.986742 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:23:48.986705    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:24:01.475982 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:24:01.475928    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:24:16.936654 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:24:16.936618    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:24:33.788143 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:24:33.788112    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:24:37.062359 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:24:37.062327    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 10:24:37.067233 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:24:37.067210    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 10:24:44.380531 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:24:44.380496    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:24:46.936884 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:24:46.936848    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:24:57.081170 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:24:57.081135    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:24:59.776465 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:24:59.776426    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:24:59.777766 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:24:59.777338    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq" podUID="fb74ea5d-0246-4fe2-8dcb-3363626c6018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:25:08.977000 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:25:08.976966    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:25:16.936707 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:25:16.936613    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:25:32.678800 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:25:32.678758    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:25:43.241862 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:25:43.241828    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:25:44.773073 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:25:44.773035    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:25:44.773681 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:25:44.773649    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq" podUID="fb74ea5d-0246-4fe2-8dcb-3363626c6018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:25:46.936588 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:25:46.936551    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:25:53.097781 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:25:53.097746    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:26:12.580426 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:26:12.580394    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:26:16.935475 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:26:16.935434    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:26:36.177758 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:26:36.177716    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:26:46.936364 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:26:46.936274    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:27:01.621174 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:27:01.621119    4473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 10:27:01.621752 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:27:01.621220    4473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl" podUID="1b80f43c-c1d9-4f07-84eb-7aa145b3d03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 10:27:16.936462 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:27:16.936426    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:27:46.936702 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:27:46.936655    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:28:16.936272 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:28:16.936189    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:28:33.972970 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:28:33.972917    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xbn6s"]
Nov 03 10:28:46.935910 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:28:46.935874    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:29:16.936167 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:29:16.936120    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:29:31.160795 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:29:31.160762    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-6vvpk_73e8a24d-31cf-44f0-99a5-5d3bd7259449/manager/0.log"
Nov 03 10:29:32.723671 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:29:32.723638    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-p72nd_dc80f365-2a40-4fd0-ae2b-6cfc815a3d63/proxy/0.log"
Nov 03 10:29:37.090114 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:29:37.090090    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 10:29:37.097369 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:29:37.097343    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 10:29:46.935661 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:29:46.935576    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:29:50.946440 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:29:50.946403    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-xbn6s_a025cb89-c80d-48f7-9f0a-93c8b6c24950/limitador/0.log"
Nov 03 10:29:52.547541 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:29:52.547510    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-x2f6z_269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6/discovery/0.log"
Nov 03 10:29:53.112284 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:29:53.112256    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-677799c6bb-fftnv_2e9796dc-2aa9-4c4b-a171-dee0cb98ca23/kube-auth-proxy/0.log"
Nov 03 10:29:53.921092 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:29:53.921061    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-9wsvs_031e6c9b-93f8-458b-a20d-24927d6fb383/istio-proxy/0.log"
Nov 03 10:29:54.236025 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:29:54.235931    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-c7mlq_fb74ea5d-0246-4fe2-8dcb-3363626c6018/istio-proxy/0.log"
Nov 03 10:29:55.137831 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:29:55.137802    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-gvlpl_1b80f43c-c1d9-4f07-84eb-7aa145b3d03c/istio-proxy/0.log"
Nov 03 10:30:09.881799 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:09.881765    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b4k9m/must-gather-7xr4v"]
Nov 03 10:30:09.882319 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:09.882143    4473 cpu_manager.go:414] "containerMap: removing container" podUID="f7dc7ea7-0abf-4f06-89dc-e79520dcd889" containerName="main"
Nov 03 10:30:09.882319 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:09.882155    4473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7dc7ea7-0abf-4f06-89dc-e79520dcd889" containerName="main"
Nov 03 10:30:09.882319 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:09.882221    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7dc7ea7-0abf-4f06-89dc-e79520dcd889" containerName="main"
Nov 03 10:30:09.882319 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:09.882232    4473 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1f33b3d-b144-463c-abe9-319e2fe0568f" containerName="cleanup"
Nov 03 10:30:09.885247 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:09.885231    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b4k9m/must-gather-7xr4v"
Nov 03 10:30:09.916298 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:09.916265    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4fpj\" (UniqueName: \"kubernetes.io/projected/c8a2c69e-6187-4898-bc9a-bf60b49fd734-kube-api-access-t4fpj\") pod \"must-gather-7xr4v\" (UID: \"c8a2c69e-6187-4898-bc9a-bf60b49fd734\") " pod="openshift-must-gather-b4k9m/must-gather-7xr4v"
Nov 03 10:30:09.916452 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:09.916312    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c8a2c69e-6187-4898-bc9a-bf60b49fd734-must-gather-output\") pod \"must-gather-7xr4v\" (UID: \"c8a2c69e-6187-4898-bc9a-bf60b49fd734\") " pod="openshift-must-gather-b4k9m/must-gather-7xr4v"
Nov 03 10:30:09.942376 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:09.942348    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b4k9m/must-gather-7xr4v"]
Nov 03 10:30:09.990358 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:09.990335    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b4k9m\"/\"openshift-service-ca.crt\""
Nov 03 10:30:09.990358 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:09.990343    4473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-b4k9m\"/\"default-dockercfg-xt4bq\""
Nov 03 10:30:09.990547 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:09.990370    4473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b4k9m\"/\"kube-root-ca.crt\""
Nov 03 10:30:10.017015 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:10.016988    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4fpj\" (UniqueName: \"kubernetes.io/projected/c8a2c69e-6187-4898-bc9a-bf60b49fd734-kube-api-access-t4fpj\") pod \"must-gather-7xr4v\" (UID: \"c8a2c69e-6187-4898-bc9a-bf60b49fd734\") " pod="openshift-must-gather-b4k9m/must-gather-7xr4v"
Nov 03 10:30:10.017147 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:10.017031    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c8a2c69e-6187-4898-bc9a-bf60b49fd734-must-gather-output\") pod \"must-gather-7xr4v\" (UID: \"c8a2c69e-6187-4898-bc9a-bf60b49fd734\") " pod="openshift-must-gather-b4k9m/must-gather-7xr4v"
Nov 03 10:30:10.017331 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:10.017317    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c8a2c69e-6187-4898-bc9a-bf60b49fd734-must-gather-output\") pod \"must-gather-7xr4v\" (UID: \"c8a2c69e-6187-4898-bc9a-bf60b49fd734\") " pod="openshift-must-gather-b4k9m/must-gather-7xr4v"
Nov 03 10:30:10.076401 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:10.076367    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4fpj\" (UniqueName: \"kubernetes.io/projected/c8a2c69e-6187-4898-bc9a-bf60b49fd734-kube-api-access-t4fpj\") pod \"must-gather-7xr4v\" (UID: \"c8a2c69e-6187-4898-bc9a-bf60b49fd734\") " pod="openshift-must-gather-b4k9m/must-gather-7xr4v"
Nov 03 10:30:10.198464 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:10.198402    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b4k9m/must-gather-7xr4v"
Nov 03 10:30:10.568887 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:10.568855    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b4k9m/must-gather-7xr4v"]
Nov 03 10:30:10.570979 ip-10-0-139-73 kubenswrapper[4473]: W1103 10:30:10.570928    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8a2c69e_6187_4898_bc9a_bf60b49fd734.slice/crio-849c1c95557e0e808afe748e4cd8bb6226a147b3de0e6892edcbdcabc0d26523 WatchSource:0}: Error finding container 849c1c95557e0e808afe748e4cd8bb6226a147b3de0e6892edcbdcabc0d26523: Status 404 returned error can't find the container with id 849c1c95557e0e808afe748e4cd8bb6226a147b3de0e6892edcbdcabc0d26523
Nov 03 10:30:10.572763 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:10.572746    4473 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 10:30:10.720891 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:10.720844    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b4k9m/must-gather-7xr4v" event={"ID":"c8a2c69e-6187-4898-bc9a-bf60b49fd734","Type":"ContainerStarted","Data":"849c1c95557e0e808afe748e4cd8bb6226a147b3de0e6892edcbdcabc0d26523"}
Nov 03 10:30:12.737521 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:12.737482    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b4k9m/must-gather-7xr4v" event={"ID":"c8a2c69e-6187-4898-bc9a-bf60b49fd734","Type":"ContainerStarted","Data":"2539e92c744029ec65d99d74e6d9c6b97393fa7893e3f329b39125e3f07400a9"}
Nov 03 10:30:12.737935 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:12.737536    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b4k9m/must-gather-7xr4v" event={"ID":"c8a2c69e-6187-4898-bc9a-bf60b49fd734","Type":"ContainerStarted","Data":"7e0a6349af8b3b152c78a6c701009def17c6cbfa9a63dcb238d16dd5a5798a7a"}
Nov 03 10:30:12.903569 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:12.903504    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-b4k9m/must-gather-7xr4v" podStartSLOduration=2.785605342 podStartE2EDuration="3.903478732s" totalImagesPullingTime="1.11787339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 10:30:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:30:12.852346007 +0000 UTC m=+6637.121409441" watchObservedRunningTime="2026-11-03 10:30:12.903478732 +0000 UTC m=+6637.172542169"
Nov 03 10:30:16.939128 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:30:16.939086    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:30:19.873666 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:19.873633    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-x9jpw_906c75b6-325c-4e89-8a10-e4e355884133/global-pull-secret-syncer/0.log"
Nov 03 10:30:22.342071 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:22.342041    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bq5p4_e1386760-3800-418f-bb1b-20e19880620b/konnectivity-agent/0.log"
Nov 03 10:30:23.975634 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:23.975599    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-6vvpk_73e8a24d-31cf-44f0-99a5-5d3bd7259449/manager/0.log"
Nov 03 10:30:24.381828 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:24.381785    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-73.us-west-1.compute.internal_0f8947715126d508f609643819d8b2a7/haproxy/0.log"
Nov 03 10:30:25.376573 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:25.376528    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-p72nd_dc80f365-2a40-4fd0-ae2b-6cfc815a3d63/proxy/0.log"
Nov 03 10:30:30.016511 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:30.016476    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-6vvpk_73e8a24d-31cf-44f0-99a5-5d3bd7259449/manager/0.log"
Nov 03 10:30:31.421435 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:31.421405    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-p72nd_dc80f365-2a40-4fd0-ae2b-6cfc815a3d63/proxy/0.log"
Nov 03 10:30:36.129280 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:36.129211    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-6vvpk_73e8a24d-31cf-44f0-99a5-5d3bd7259449/manager/0.log"
Nov 03 10:30:37.518633 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:37.518604    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-p72nd_dc80f365-2a40-4fd0-ae2b-6cfc815a3d63/proxy/0.log"
Nov 03 10:30:42.431354 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:42.431316    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-x2f6z_269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6/discovery/0.log"
Nov 03 10:30:42.758549 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:42.758472    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-677799c6bb-fftnv_2e9796dc-2aa9-4c4b-a171-dee0cb98ca23/kube-auth-proxy/0.log"
Nov 03 10:30:43.418811 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:43.418776    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-9wsvs_031e6c9b-93f8-458b-a20d-24927d6fb383/istio-proxy/0.log"
Nov 03 10:30:43.636005 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:43.635973    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-c7mlq_fb74ea5d-0246-4fe2-8dcb-3363626c6018/istio-proxy/0.log"
Nov 03 10:30:44.290252 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:44.290224    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-gvlpl_1b80f43c-c1d9-4f07-84eb-7aa145b3d03c/istio-proxy/0.log"
Nov 03 10:30:46.936594 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:30:46.936555    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:30:49.757320 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:49.757287    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-xbn6s_a025cb89-c80d-48f7-9f0a-93c8b6c24950/limitador/0.log"
Nov 03 10:30:51.906514 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:51.906486    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-xbn6s_a025cb89-c80d-48f7-9f0a-93c8b6c24950/limitador/0.log"
Nov 03 10:30:53.107219 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:53.107188    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e5f3c403-9baf-4ac8-8524-88513a5b7d1c/alertmanager/0.log"
Nov 03 10:30:53.531384 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:53.531354    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e5f3c403-9baf-4ac8-8524-88513a5b7d1c/config-reloader/0.log"
Nov 03 10:30:53.955540 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:53.955507    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e5f3c403-9baf-4ac8-8524-88513a5b7d1c/kube-rbac-proxy-web/0.log"
Nov 03 10:30:54.381193 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:54.381161    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e5f3c403-9baf-4ac8-8524-88513a5b7d1c/kube-rbac-proxy/0.log"
Nov 03 10:30:54.805074 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:54.805005    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e5f3c403-9baf-4ac8-8524-88513a5b7d1c/kube-rbac-proxy-metric/0.log"
Nov 03 10:30:55.229488 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:55.229454    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e5f3c403-9baf-4ac8-8524-88513a5b7d1c/prom-label-proxy/0.log"
Nov 03 10:30:55.653770 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:55.653740    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e5f3c403-9baf-4ac8-8524-88513a5b7d1c/init-config-reloader/0.log"
Nov 03 10:30:57.008658 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:57.008622    4473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b4k9m/perf-node-gather-daemonset-qq98c"]
Nov 03 10:30:57.012655 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:57.012634    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-qq98c"
Nov 03 10:30:57.057253 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:57.057223    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/77c24b43-5d01-40ff-a68f-0da08f7a3653-lib-modules\") pod \"perf-node-gather-daemonset-qq98c\" (UID: \"77c24b43-5d01-40ff-a68f-0da08f7a3653\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-qq98c"
Nov 03 10:30:57.057403 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:57.057275    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psd7v\" (UniqueName: \"kubernetes.io/projected/77c24b43-5d01-40ff-a68f-0da08f7a3653-kube-api-access-psd7v\") pod \"perf-node-gather-daemonset-qq98c\" (UID: \"77c24b43-5d01-40ff-a68f-0da08f7a3653\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-qq98c"
Nov 03 10:30:57.057403 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:57.057352    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/77c24b43-5d01-40ff-a68f-0da08f7a3653-proc\") pod \"perf-node-gather-daemonset-qq98c\" (UID: \"77c24b43-5d01-40ff-a68f-0da08f7a3653\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-qq98c"
Nov 03 10:30:57.057403 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:57.057396    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77c24b43-5d01-40ff-a68f-0da08f7a3653-sys\") pod \"perf-node-gather-daemonset-qq98c\" (UID: \"77c24b43-5d01-40ff-a68f-0da08f7a3653\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-qq98c"
Nov 03 10:30:57.057521 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:57.057416    4473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/77c24b43-5d01-40ff-a68f-0da08f7a3653-podres\") pod \"perf-node-gather-daemonset-qq98c\" (UID: \"77c24b43-5d01-40ff-a68f-0da08f7a3653\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-qq98c"
Nov 03 10:30:57.102191 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:57.102163    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b4k9m/perf-node-gather-daemonset-qq98c"]
Nov 03 10:30:57.158682 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:57.158654    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psd7v\" (UniqueName: \"kubernetes.io/projected/77c24b43-5d01-40ff-a68f-0da08f7a3653-kube-api-access-psd7v\") pod \"perf-node-gather-daemonset-qq98c\" (UID: \"77c24b43-5d01-40ff-a68f-0da08f7a3653\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-qq98c"
Nov 03 10:30:57.158827 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:57.158695    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/77c24b43-5d01-40ff-a68f-0da08f7a3653-proc\") pod \"perf-node-gather-daemonset-qq98c\" (UID: \"77c24b43-5d01-40ff-a68f-0da08f7a3653\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-qq98c"
Nov 03 10:30:57.158827 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:57.158733    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77c24b43-5d01-40ff-a68f-0da08f7a3653-sys\") pod \"perf-node-gather-daemonset-qq98c\" (UID: \"77c24b43-5d01-40ff-a68f-0da08f7a3653\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-qq98c"
Nov 03 10:30:57.158827 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:57.158794    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77c24b43-5d01-40ff-a68f-0da08f7a3653-sys\") pod \"perf-node-gather-daemonset-qq98c\" (UID: \"77c24b43-5d01-40ff-a68f-0da08f7a3653\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-qq98c"
Nov 03 10:30:57.158827 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:57.158800    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/77c24b43-5d01-40ff-a68f-0da08f7a3653-proc\") pod \"perf-node-gather-daemonset-qq98c\" (UID: \"77c24b43-5d01-40ff-a68f-0da08f7a3653\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-qq98c"
Nov 03 10:30:57.158827 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:57.158822    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/77c24b43-5d01-40ff-a68f-0da08f7a3653-podres\") pod \"perf-node-gather-daemonset-qq98c\" (UID: \"77c24b43-5d01-40ff-a68f-0da08f7a3653\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-qq98c"
Nov 03 10:30:57.159035 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:57.158878    4473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/77c24b43-5d01-40ff-a68f-0da08f7a3653-lib-modules\") pod \"perf-node-gather-daemonset-qq98c\" (UID: \"77c24b43-5d01-40ff-a68f-0da08f7a3653\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-qq98c"
Nov 03 10:30:57.159035 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:57.158933    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/77c24b43-5d01-40ff-a68f-0da08f7a3653-podres\") pod \"perf-node-gather-daemonset-qq98c\" (UID: \"77c24b43-5d01-40ff-a68f-0da08f7a3653\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-qq98c"
Nov 03 10:30:57.159035 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:57.158987    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/77c24b43-5d01-40ff-a68f-0da08f7a3653-lib-modules\") pod \"perf-node-gather-daemonset-qq98c\" (UID: \"77c24b43-5d01-40ff-a68f-0da08f7a3653\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-qq98c"
Nov 03 10:30:57.217262 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:57.217233    4473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psd7v\" (UniqueName: \"kubernetes.io/projected/77c24b43-5d01-40ff-a68f-0da08f7a3653-kube-api-access-psd7v\") pod \"perf-node-gather-daemonset-qq98c\" (UID: \"77c24b43-5d01-40ff-a68f-0da08f7a3653\") " pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-qq98c"
Nov 03 10:30:57.325559 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:57.325502    4473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-qq98c"
Nov 03 10:30:57.694093 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:57.694072    4473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b4k9m/perf-node-gather-daemonset-qq98c"]
Nov 03 10:30:57.696354 ip-10-0-139-73 kubenswrapper[4473]: W1103 10:30:57.696328    4473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod77c24b43_5d01_40ff_a68f_0da08f7a3653.slice/crio-d7e947aa7e5071adda18fa9ccc55a80f3c1bd8c88a091edf4bcee12914532929 WatchSource:0}: Error finding container d7e947aa7e5071adda18fa9ccc55a80f3c1bd8c88a091edf4bcee12914532929: Status 404 returned error can't find the container with id d7e947aa7e5071adda18fa9ccc55a80f3c1bd8c88a091edf4bcee12914532929
Nov 03 10:30:57.873531 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:57.873504    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-748b854566-dbwqz_809f0f9a-ccb4-4bca-b6dc-1af7b9a66555/metrics-server/0.log"
Nov 03 10:30:57.976170 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:57.976092    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-qq98c" event={"ID":"77c24b43-5d01-40ff-a68f-0da08f7a3653","Type":"ContainerStarted","Data":"8c2644e6f3fb4bd666434b9b0589d0e0353726a4c67b15c31f7f840a6ceee487"}
Nov 03 10:30:57.976170 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:57.976142    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-qq98c" event={"ID":"77c24b43-5d01-40ff-a68f-0da08f7a3653","Type":"ContainerStarted","Data":"d7e947aa7e5071adda18fa9ccc55a80f3c1bd8c88a091edf4bcee12914532929"}
Nov 03 10:30:57.976418 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:57.976274    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-qq98c"
Nov 03 10:30:58.092130 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:58.092076    4473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-qq98c" podStartSLOduration=2.092062611 podStartE2EDuration="2.092062611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 10:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 10:30:58.090571071 +0000 UTC m=+6682.359634506" watchObservedRunningTime="2026-11-03 10:30:58.092062611 +0000 UTC m=+6682.361126077"
Nov 03 10:30:58.300206 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:30:58.300048    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-4dvrv_024a7379-f2b5-41ce-95a0-58da9ad664fb/monitoring-plugin/0.log"
Nov 03 10:31:00.005291 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:00.005261    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hjsfs_ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9/node-exporter/0.log"
Nov 03 10:31:00.428526 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:00.428494    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hjsfs_ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9/kube-rbac-proxy/0.log"
Nov 03 10:31:00.852987 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:00.852891    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hjsfs_ba36bbf3-f9d7-4f0b-98af-ee5dc6fe87e9/init-textfile/0.log"
Nov 03 10:31:03.992494 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:03.992467    4473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-b4k9m/perf-node-gather-daemonset-qq98c"
Nov 03 10:31:06.805909 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:06.805878    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-qrl96_b6b8ee0a-1be5-40c0-8878-4eba660f6309/prometheus-operator/0.log"
Nov 03 10:31:07.266592 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:07.266564    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-qrl96_b6b8ee0a-1be5-40c0-8878-4eba660f6309/kube-rbac-proxy/0.log"
Nov 03 10:31:07.695895 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:07.695860    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-pbpxk_ae525063-a06b-4860-af1a-3bcd116858c6/prometheus-operator-admission-webhook/0.log"
Nov 03 10:31:08.125846 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:08.125825    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6d8c96b47-jh99f_bfbbe91f-114b-4e33-a356-511602c939e7/telemeter-client/0.log"
Nov 03 10:31:08.551877 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:08.551794    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6d8c96b47-jh99f_bfbbe91f-114b-4e33-a356-511602c939e7/reload/0.log"
Nov 03 10:31:08.976268 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:08.976243    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6d8c96b47-jh99f_bfbbe91f-114b-4e33-a356-511602c939e7/kube-rbac-proxy/0.log"
Nov 03 10:31:09.407143 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:09.407110    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c5548d8cc-xj2td_b030cb9d-3c84-4482-9761-a507e84c9204/thanos-query/0.log"
Nov 03 10:31:09.832583 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:09.832489    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c5548d8cc-xj2td_b030cb9d-3c84-4482-9761-a507e84c9204/kube-rbac-proxy-web/0.log"
Nov 03 10:31:10.256742 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:10.256700    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c5548d8cc-xj2td_b030cb9d-3c84-4482-9761-a507e84c9204/kube-rbac-proxy/0.log"
Nov 03 10:31:10.680205 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:10.680170    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c5548d8cc-xj2td_b030cb9d-3c84-4482-9761-a507e84c9204/prom-label-proxy/0.log"
Nov 03 10:31:11.104429 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:11.104345    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c5548d8cc-xj2td_b030cb9d-3c84-4482-9761-a507e84c9204/kube-rbac-proxy-rules/0.log"
Nov 03 10:31:11.527195 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:11.527165    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c5548d8cc-xj2td_b030cb9d-3c84-4482-9761-a507e84c9204/kube-rbac-proxy-metrics/0.log"
Nov 03 10:31:16.936784 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:31:16.936712    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:31:18.318128 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:18.318106    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/cert-manager_cert-manager-webhook-5dbbbf75cc-8p78z_6882d8dd-2f29-4648-bc54-42f039a393e5/cert-manager-webhook/0.log"
Nov 03 10:31:19.606834 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:19.606794    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5bf4b887bc-p5l6b_a18a1e24-c5b9-4328-952a-e235d0249337/console/0.log"
Nov 03 10:31:21.473859 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:21.473828    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-77df88dcb4-r9mf2_1aede497-0548-4f08-9e87-f24e89e735fe/manager/0.log"
Nov 03 10:31:21.760386 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:21.760315    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-77df88dcb4-r9mf2_1aede497-0548-4f08-9e87-f24e89e735fe/copy-manifests/0.log"
Nov 03 10:31:25.679320 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:25.679285    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-6vvpk_73e8a24d-31cf-44f0-99a5-5d3bd7259449/manager/0.log"
Nov 03 10:31:27.051394 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:27.051363    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-p72nd_dc80f365-2a40-4fd0-ae2b-6cfc815a3d63/proxy/0.log"
Nov 03 10:31:30.011626 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:30.011599    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gvbpd_9e8945a0-1bd9-4179-8996-a165bb1db1c4/dns/0.log"
Nov 03 10:31:30.434551 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:30.434519    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gvbpd_9e8945a0-1bd9-4179-8996-a165bb1db1c4/kube-rbac-proxy/0.log"
Nov 03 10:31:30.602691 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:30.602665    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-x2f6z_269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6/discovery/0.log"
Nov 03 10:31:30.944336 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:30.944310    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-677799c6bb-fftnv_2e9796dc-2aa9-4c4b-a171-dee0cb98ca23/kube-auth-proxy/0.log"
Nov 03 10:31:31.603395 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:31.603368    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-9wsvs_031e6c9b-93f8-458b-a20d-24927d6fb383/istio-proxy/0.log"
Nov 03 10:31:31.707661 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:31.707635    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hgbb5_801881d6-f217-4c80-a190-af953df3af04/dns-node-resolver/0.log"
Nov 03 10:31:31.820922 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:31.820890    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-c7mlq_fb74ea5d-0246-4fe2-8dcb-3363626c6018/istio-proxy/0.log"
Nov 03 10:31:32.474685 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:32.474652    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-gvlpl_1b80f43c-c1d9-4f07-84eb-7aa145b3d03c/istio-proxy/0.log"
Nov 03 10:31:35.114769 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:35.114738    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5drgw_110651f3-9a70-43bf-888d-0813682bc891/node-ca/0.log"
Nov 03 10:31:41.832845 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:41.832806    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-x2f6z_269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6/discovery/0.log"
Nov 03 10:31:42.175644 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:42.175612    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-677799c6bb-fftnv_2e9796dc-2aa9-4c4b-a171-dee0cb98ca23/kube-auth-proxy/0.log"
Nov 03 10:31:42.314813 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:42.314782    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-x2f6z_269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6/discovery/0.log"
Nov 03 10:31:42.889808 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:42.889775    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-9wsvs_031e6c9b-93f8-458b-a20d-24927d6fb383/istio-proxy/0.log"
Nov 03 10:31:43.101120 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:43.101089    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-c7mlq_fb74ea5d-0246-4fe2-8dcb-3363626c6018/istio-proxy/0.log"
Nov 03 10:31:43.753168 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:43.753077    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-gvlpl_1b80f43c-c1d9-4f07-84eb-7aa145b3d03c/istio-proxy/0.log"
Nov 03 10:31:44.011059 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:44.010988    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-677799c6bb-fftnv_2e9796dc-2aa9-4c4b-a171-dee0cb98ca23/kube-auth-proxy/0.log"
Nov 03 10:31:45.293236 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:45.293203    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-9wsvs_031e6c9b-93f8-458b-a20d-24927d6fb383/istio-proxy/0.log"
Nov 03 10:31:45.719963 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:45.719927    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-c7mlq_fb74ea5d-0246-4fe2-8dcb-3363626c6018/istio-proxy/0.log"
Nov 03 10:31:46.935920 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:31:46.935882    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:31:47.011895 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:47.011859    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-gvlpl_1b80f43c-c1d9-4f07-84eb-7aa145b3d03c/istio-proxy/0.log"
Nov 03 10:31:51.477638 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:51.477600    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-x2f6z_269aa896-4fbc-4cdb-ac6e-1f2978fc6bd6/discovery/0.log"
Nov 03 10:31:51.819988 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:51.819890    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-677799c6bb-fftnv_2e9796dc-2aa9-4c4b-a171-dee0cb98ca23/kube-auth-proxy/0.log"
Nov 03 10:31:52.481444 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:52.481413    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-9wsvs_031e6c9b-93f8-458b-a20d-24927d6fb383/istio-proxy/0.log"
Nov 03 10:31:52.708480 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:52.708451    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-c7mlq_fb74ea5d-0246-4fe2-8dcb-3363626c6018/istio-proxy/0.log"
Nov 03 10:31:53.367523 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:53.367498    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-gvlpl_1b80f43c-c1d9-4f07-84eb-7aa145b3d03c/istio-proxy/0.log"
Nov 03 10:31:54.431660 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:31:54.431633    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-74dwn_d7329659-0fc1-4d97-b321-11e90bf7d13c/serve-healthcheck-canary/0.log"
Nov 03 10:32:00.301787 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:32:00.301755    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tft2z_49fbaf82-5d99-48e1-90eb-005aec9f3d1b/kube-rbac-proxy/0.log"
Nov 03 10:32:00.724990 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:32:00.724939    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tft2z_49fbaf82-5d99-48e1-90eb-005aec9f3d1b/exporter/0.log"
Nov 03 10:32:01.148860 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:32:01.148830    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tft2z_49fbaf82-5d99-48e1-90eb-005aec9f3d1b/extractor/0.log"
Nov 03 10:32:08.085714 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:32:08.085687    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-77df88dcb4-r9mf2_1aede497-0548-4f08-9e87-f24e89e735fe/manager/0.log"
Nov 03 10:32:08.638758 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:32:08.638727    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-77df88dcb4-r9mf2_1aede497-0548-4f08-9e87-f24e89e735fe/copy-manifests/0.log"
Nov 03 10:32:15.943439 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:32:15.943405    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-xbn6s_a025cb89-c80d-48f7-9f0a-93c8b6c24950/limitador/0.log"
Nov 03 10:32:16.717255 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:32:16.717222    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-6vvpk_73e8a24d-31cf-44f0-99a5-5d3bd7259449/manager/0.log"
Nov 03 10:32:16.937626 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:32:16.937591    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:32:19.646245 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:32:19.646220    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-p72nd_dc80f365-2a40-4fd0-ae2b-6cfc815a3d63/proxy/0.log"
Nov 03 10:32:46.937250 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:32:46.937164    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:32:51.836309 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:32:51.836280    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-xbn6s_a025cb89-c80d-48f7-9f0a-93c8b6c24950/limitador/0.log"
Nov 03 10:32:54.216007 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:32:54.215975    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wsrmr_968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9/kube-multus-additional-cni-plugins/0.log"
Nov 03 10:32:54.638487 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:32:54.638463    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wsrmr_968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9/egress-router-binary-copy/0.log"
Nov 03 10:32:55.062311 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:32:55.062240    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wsrmr_968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9/cni-plugins/0.log"
Nov 03 10:32:55.485070 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:32:55.485040    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wsrmr_968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9/bond-cni-plugin/0.log"
Nov 03 10:32:55.911383 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:32:55.911353    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wsrmr_968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9/routeoverride-cni/0.log"
Nov 03 10:32:56.334805 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:32:56.334732    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wsrmr_968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9/whereabouts-cni-bincopy/0.log"
Nov 03 10:32:56.756873 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:32:56.756837    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wsrmr_968e5615-5e12-4dbf-b14d-e1fdb1e1cbe9/whereabouts-cni/0.log"
Nov 03 10:32:57.238579 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:32:57.238543    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-kzcqg_105e26de-f1f1-4aae-99ef-e6a9491ccbe8/kube-multus/0.log"
Nov 03 10:32:58.659413 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:32:58.659370    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-pflts_7c26dfa4-e47e-43a1-94a2-816d22fb7f3a/network-metrics-daemon/0.log"
Nov 03 10:32:59.074997 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:32:59.074928    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-pflts_7c26dfa4-e47e-43a1-94a2-816d22fb7f3a/kube-rbac-proxy/0.log"
Nov 03 10:33:04.562856 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:33:04.562823    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-controller/0.log"
Nov 03 10:33:05.014228 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:33:05.014198    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/1.log"
Nov 03 10:33:05.014228 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:33:05.014231    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 10:33:05.332697 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:33:05.332613    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/kube-rbac-proxy-node/0.log"
Nov 03 10:33:05.756249 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:33:05.756222    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/kube-rbac-proxy-ovn-metrics/0.log"
Nov 03 10:33:06.225606 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:33:06.225577    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/northd/0.log"
Nov 03 10:33:06.667198 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:33:06.667171    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/nbdb/0.log"
Nov 03 10:33:07.092365 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:33:07.092296    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/sbdb/0.log"
Nov 03 10:33:07.617520 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:33:07.617494    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovnkube-controller/0.log"
Nov 03 10:33:16.939208 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:33:16.939172    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:33:21.816376 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:33:21.816342    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-clx84_f6ca3dc3-231e-4a8d-a16c-7384c225110a/network-check-target-container/0.log"
Nov 03 10:33:28.013089 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:33:28.013063    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-xbn6s_a025cb89-c80d-48f7-9f0a-93c8b6c24950/limitador/0.log"
Nov 03 10:33:29.309572 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:33:29.309542    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-689v8_6b529371-cf9d-40e4-a2c5-55d69ab9a08d/iptables-alerter/0.log"
Nov 03 10:33:34.933574 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:33:34.933531    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-jpll8_d1649ad9-921c-49d9-baa5-c51f44175b8c/tuned/0.log"
Nov 03 10:33:46.936563 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:33:46.936523    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:33:59.117603 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:33:59.117572    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-l8cfs_7e02a4de-6df2-4930-8360-4c5e09897472/csi-driver/0.log"
Nov 03 10:33:59.541430 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:33:59.541357    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-l8cfs_7e02a4de-6df2-4930-8360-4c5e09897472/csi-node-driver-registrar/0.log"
Nov 03 10:33:59.963988 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:33:59.963938    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-l8cfs_7e02a4de-6df2-4930-8360-4c5e09897472/csi-liveness-probe/0.log"
Nov 03 10:34:04.385453 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:04.385427    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-xbn6s_a025cb89-c80d-48f7-9f0a-93c8b6c24950/limitador/0.log"
Nov 03 10:34:16.938315 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:34:16.938279    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:34:21.989995 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:21.989957    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"]
Nov 03 10:34:21.990414 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:21.990150    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs" podUID="031e6c9b-93f8-458b-a20d-24927d6fb383" containerName="istio-proxy" containerID="cri-o://67004f48df7494bae6905dac1007d5233f73b00c7808dad8610a54665aed4321" gracePeriod=30
Nov 03 10:34:22.038681 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:22.038653    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"]
Nov 03 10:34:22.038867 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:22.038847    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl" podUID="1b80f43c-c1d9-4f07-84eb-7aa145b3d03c" containerName="istio-proxy" containerID="cri-o://7e0e27efcf2d0388eb1dc4d9013c9b575739d7fd33a1f1e391294b6ab2775852" gracePeriod=30
Nov 03 10:34:22.039905 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:22.039880    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"]
Nov 03 10:34:22.040111 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:22.040055    4473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq" podUID="fb74ea5d-0246-4fe2-8dcb-3363626c6018" containerName="istio-proxy" containerID="cri-o://68c7cc705b268144f5d8c1d337e4d099257f0bc95573461fbc9ef043101602ed" gracePeriod=30
Nov 03 10:34:27.293634 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.293609    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"
Nov 03 10:34:27.318698 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.318675    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"
Nov 03 10:34:27.326541 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.326521    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"
Nov 03 10:34:27.449724 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.449692    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-kube-api-access-2jn5n\" (UniqueName: \"kubernetes.io/projected/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-kube-api-access-2jn5n\") pod \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\" (UID: \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\") "
Nov 03 10:34:27.449724 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.449728    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb74ea5d-0246-4fe2-8dcb-3363626c6018-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb74ea5d-0246-4fe2-8dcb-3363626c6018-istiod-ca-cert\") pod \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\" (UID: \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\") "
Nov 03 10:34:27.449985 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.449753    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/031e6c9b-93f8-458b-a20d-24927d6fb383-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/031e6c9b-93f8-458b-a20d-24927d6fb383-istio-data\") pod \"031e6c9b-93f8-458b-a20d-24927d6fb383\" (UID: \"031e6c9b-93f8-458b-a20d-24927d6fb383\") "
Nov 03 10:34:27.449985 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.449770    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/031e6c9b-93f8-458b-a20d-24927d6fb383-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/031e6c9b-93f8-458b-a20d-24927d6fb383-workload-socket\") pod \"031e6c9b-93f8-458b-a20d-24927d6fb383\" (UID: \"031e6c9b-93f8-458b-a20d-24927d6fb383\") "
Nov 03 10:34:27.449985 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.449900    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-workload-certs\") pod \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\" (UID: \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\") "
Nov 03 10:34:27.449985 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.449940    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/031e6c9b-93f8-458b-a20d-24927d6fb383-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/031e6c9b-93f8-458b-a20d-24927d6fb383-workload-certs\") pod \"031e6c9b-93f8-458b-a20d-24927d6fb383\" (UID: \"031e6c9b-93f8-458b-a20d-24927d6fb383\") "
Nov 03 10:34:27.450230 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.449997    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb74ea5d-0246-4fe2-8dcb-3363626c6018-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea5d-0246-4fe2-8dcb-3363626c6018-workload-socket\") pod \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\" (UID: \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\") "
Nov 03 10:34:27.450230 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.450035    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-istio-token\" (UniqueName: \"kubernetes.io/projected/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-istio-token\") pod \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\" (UID: \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\") "
Nov 03 10:34:27.450230 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.450042    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/031e6c9b-93f8-458b-a20d-24927d6fb383-workload-socket" pod "031e6c9b-93f8-458b-a20d-24927d6fb383" (UID: "031e6c9b-93f8-458b-a20d-24927d6fb383"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:34:27.450230 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.450053    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/031e6c9b-93f8-458b-a20d-24927d6fb383-istio-data" pod "031e6c9b-93f8-458b-a20d-24927d6fb383" (UID: "031e6c9b-93f8-458b-a20d-24927d6fb383"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:34:27.450230 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.450081    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-istio-data\") pod \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\" (UID: \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\") "
Nov 03 10:34:27.450230 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.450106    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-istiod-ca-cert\") pod \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\" (UID: \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\") "
Nov 03 10:34:27.450230 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.450112    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb74ea5d-0246-4fe2-8dcb-3363626c6018-istiod-ca-cert" pod "fb74ea5d-0246-4fe2-8dcb-3363626c6018" (UID: "fb74ea5d-0246-4fe2-8dcb-3363626c6018"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:34:27.450230 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.450135    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-istio-podinfo\") pod \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\" (UID: \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\") "
Nov 03 10:34:27.450230 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.450176    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/031e6c9b-93f8-458b-a20d-24927d6fb383-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/031e6c9b-93f8-458b-a20d-24927d6fb383-istiod-ca-cert\") pod \"031e6c9b-93f8-458b-a20d-24927d6fb383\" (UID: \"031e6c9b-93f8-458b-a20d-24927d6fb383\") "
Nov 03 10:34:27.450230 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.450205    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb74ea5d-0246-4fe2-8dcb-3363626c6018-kube-api-access-9dfqv\" (UniqueName: \"kubernetes.io/projected/fb74ea5d-0246-4fe2-8dcb-3363626c6018-kube-api-access-9dfqv\") pod \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\" (UID: \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\") "
Nov 03 10:34:27.450842 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.450246    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb74ea5d-0246-4fe2-8dcb-3363626c6018-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea5d-0246-4fe2-8dcb-3363626c6018-credential-socket\") pod \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\" (UID: \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\") "
Nov 03 10:34:27.450842 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.450239    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-workload-certs" pod "1b80f43c-c1d9-4f07-84eb-7aa145b3d03c" (UID: "1b80f43c-c1d9-4f07-84eb-7aa145b3d03c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:34:27.450842 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.450279    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb74ea5d-0246-4fe2-8dcb-3363626c6018-workload-socket" pod "fb74ea5d-0246-4fe2-8dcb-3363626c6018" (UID: "fb74ea5d-0246-4fe2-8dcb-3363626c6018"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:34:27.450842 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.450289    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb74ea5d-0246-4fe2-8dcb-3363626c6018-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea5d-0246-4fe2-8dcb-3363626c6018-istio-envoy\") pod \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\" (UID: \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\") "
Nov 03 10:34:27.450842 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.450327    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb74ea5d-0246-4fe2-8dcb-3363626c6018-istio-token\" (UniqueName: \"kubernetes.io/projected/fb74ea5d-0246-4fe2-8dcb-3363626c6018-istio-token\") pod \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\" (UID: \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\") "
Nov 03 10:34:27.450842 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.450367    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/031e6c9b-93f8-458b-a20d-24927d6fb383-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/031e6c9b-93f8-458b-a20d-24927d6fb383-istio-podinfo\") pod \"031e6c9b-93f8-458b-a20d-24927d6fb383\" (UID: \"031e6c9b-93f8-458b-a20d-24927d6fb383\") "
Nov 03 10:34:27.450842 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.450405    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb74ea5d-0246-4fe2-8dcb-3363626c6018-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea5d-0246-4fe2-8dcb-3363626c6018-istio-data\") pod \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\" (UID: \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\") "
Nov 03 10:34:27.450842 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.450436    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/031e6c9b-93f8-458b-a20d-24927d6fb383-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/031e6c9b-93f8-458b-a20d-24927d6fb383-credential-socket\") pod \"031e6c9b-93f8-458b-a20d-24927d6fb383\" (UID: \"031e6c9b-93f8-458b-a20d-24927d6fb383\") "
Nov 03 10:34:27.450842 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.450477    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/031e6c9b-93f8-458b-a20d-24927d6fb383-istio-token\" (UniqueName: \"kubernetes.io/projected/031e6c9b-93f8-458b-a20d-24927d6fb383-istio-token\") pod \"031e6c9b-93f8-458b-a20d-24927d6fb383\" (UID: \"031e6c9b-93f8-458b-a20d-24927d6fb383\") "
Nov 03 10:34:27.450842 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.450507    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/031e6c9b-93f8-458b-a20d-24927d6fb383-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/031e6c9b-93f8-458b-a20d-24927d6fb383-istio-envoy\") pod \"031e6c9b-93f8-458b-a20d-24927d6fb383\" (UID: \"031e6c9b-93f8-458b-a20d-24927d6fb383\") "
Nov 03 10:34:27.450842 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.450551    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-workload-socket\") pod \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\" (UID: \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\") "
Nov 03 10:34:27.450842 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.450578    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-istio-envoy\") pod \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\" (UID: \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\") "
Nov 03 10:34:27.450842 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.450614    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-credential-socket\") pod \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\" (UID: \"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c\") "
Nov 03 10:34:27.450842 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.450652    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/031e6c9b-93f8-458b-a20d-24927d6fb383-kube-api-access-b2jtg\" (UniqueName: \"kubernetes.io/projected/031e6c9b-93f8-458b-a20d-24927d6fb383-kube-api-access-b2jtg\") pod \"031e6c9b-93f8-458b-a20d-24927d6fb383\" (UID: \"031e6c9b-93f8-458b-a20d-24927d6fb383\") "
Nov 03 10:34:27.450842 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.450684    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb74ea5d-0246-4fe2-8dcb-3363626c6018-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea5d-0246-4fe2-8dcb-3363626c6018-workload-certs\") pod \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\" (UID: \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\") "
Nov 03 10:34:27.450842 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.450720    4473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb74ea5d-0246-4fe2-8dcb-3363626c6018-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb74ea5d-0246-4fe2-8dcb-3363626c6018-istio-podinfo\") pod \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\" (UID: \"fb74ea5d-0246-4fe2-8dcb-3363626c6018\") "
Nov 03 10:34:27.452890 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.451099    4473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb74ea5d-0246-4fe2-8dcb-3363626c6018-istiod-ca-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 10:34:27.452890 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.451120    4473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/031e6c9b-93f8-458b-a20d-24927d6fb383-istio-data\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 10:34:27.452890 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.451137    4473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/031e6c9b-93f8-458b-a20d-24927d6fb383-workload-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 10:34:27.452890 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.451152    4473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-workload-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 10:34:27.452890 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.451167    4473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea5d-0246-4fe2-8dcb-3363626c6018-workload-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 10:34:27.452890 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.450291    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/031e6c9b-93f8-458b-a20d-24927d6fb383-workload-certs" pod "031e6c9b-93f8-458b-a20d-24927d6fb383" (UID: "031e6c9b-93f8-458b-a20d-24927d6fb383"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:34:27.452890 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.450479    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb74ea5d-0246-4fe2-8dcb-3363626c6018-credential-socket" pod "fb74ea5d-0246-4fe2-8dcb-3363626c6018" (UID: "fb74ea5d-0246-4fe2-8dcb-3363626c6018"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:34:27.452890 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.450611    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-istio-data" pod "1b80f43c-c1d9-4f07-84eb-7aa145b3d03c" (UID: "1b80f43c-c1d9-4f07-84eb-7aa145b3d03c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:34:27.452890 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.450638    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/031e6c9b-93f8-458b-a20d-24927d6fb383-istiod-ca-cert" pod "031e6c9b-93f8-458b-a20d-24927d6fb383" (UID: "031e6c9b-93f8-458b-a20d-24927d6fb383"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:34:27.452890 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.450926    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-istiod-ca-cert" pod "1b80f43c-c1d9-4f07-84eb-7aa145b3d03c" (UID: "1b80f43c-c1d9-4f07-84eb-7aa145b3d03c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 10:34:27.452890 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.452174    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-credential-socket" pod "1b80f43c-c1d9-4f07-84eb-7aa145b3d03c" (UID: "1b80f43c-c1d9-4f07-84eb-7aa145b3d03c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:34:27.452890 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.452379    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-workload-socket" pod "1b80f43c-c1d9-4f07-84eb-7aa145b3d03c" (UID: "1b80f43c-c1d9-4f07-84eb-7aa145b3d03c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:34:27.452890 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.452549    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb74ea5d-0246-4fe2-8dcb-3363626c6018-istio-data" pod "fb74ea5d-0246-4fe2-8dcb-3363626c6018" (UID: "fb74ea5d-0246-4fe2-8dcb-3363626c6018"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:34:27.452890 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.452611    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/031e6c9b-93f8-458b-a20d-24927d6fb383-credential-socket" pod "031e6c9b-93f8-458b-a20d-24927d6fb383" (UID: "031e6c9b-93f8-458b-a20d-24927d6fb383"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:34:27.452890 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.452660    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb74ea5d-0246-4fe2-8dcb-3363626c6018-workload-certs" pod "fb74ea5d-0246-4fe2-8dcb-3363626c6018" (UID: "fb74ea5d-0246-4fe2-8dcb-3363626c6018"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:34:27.454063 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.453217    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb74ea5d-0246-4fe2-8dcb-3363626c6018-kube-api-access-9dfqv" pod "fb74ea5d-0246-4fe2-8dcb-3363626c6018" (UID: "fb74ea5d-0246-4fe2-8dcb-3363626c6018"). InnerVolumeSpecName "kube-api-access-9dfqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:34:27.454063 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.453466    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-kube-api-access-2jn5n" pod "1b80f43c-c1d9-4f07-84eb-7aa145b3d03c" (UID: "1b80f43c-c1d9-4f07-84eb-7aa145b3d03c"). InnerVolumeSpecName "kube-api-access-2jn5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:34:27.454063 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.453629    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb74ea5d-0246-4fe2-8dcb-3363626c6018-istio-token" pod "fb74ea5d-0246-4fe2-8dcb-3363626c6018" (UID: "fb74ea5d-0246-4fe2-8dcb-3363626c6018"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:34:27.454063 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.453837    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-istio-token" pod "1b80f43c-c1d9-4f07-84eb-7aa145b3d03c" (UID: "1b80f43c-c1d9-4f07-84eb-7aa145b3d03c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:34:27.454749 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.454559    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/031e6c9b-93f8-458b-a20d-24927d6fb383-istio-token" pod "031e6c9b-93f8-458b-a20d-24927d6fb383" (UID: "031e6c9b-93f8-458b-a20d-24927d6fb383"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:34:27.454749 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.454718    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-istio-podinfo" pod "1b80f43c-c1d9-4f07-84eb-7aa145b3d03c" (UID: "1b80f43c-c1d9-4f07-84eb-7aa145b3d03c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:34:27.455058 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.455031    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb74ea5d-0246-4fe2-8dcb-3363626c6018-istio-podinfo" pod "fb74ea5d-0246-4fe2-8dcb-3363626c6018" (UID: "fb74ea5d-0246-4fe2-8dcb-3363626c6018"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:34:27.455155 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.455036    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-istio-envoy" pod "1b80f43c-c1d9-4f07-84eb-7aa145b3d03c" (UID: "1b80f43c-c1d9-4f07-84eb-7aa145b3d03c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:34:27.455258 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.455241    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/031e6c9b-93f8-458b-a20d-24927d6fb383-istio-envoy" pod "031e6c9b-93f8-458b-a20d-24927d6fb383" (UID: "031e6c9b-93f8-458b-a20d-24927d6fb383"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:34:27.455425 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.455406    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/031e6c9b-93f8-458b-a20d-24927d6fb383-istio-podinfo" pod "031e6c9b-93f8-458b-a20d-24927d6fb383" (UID: "031e6c9b-93f8-458b-a20d-24927d6fb383"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 10:34:27.455512 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.455495    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb74ea5d-0246-4fe2-8dcb-3363626c6018-istio-envoy" pod "fb74ea5d-0246-4fe2-8dcb-3363626c6018" (UID: "fb74ea5d-0246-4fe2-8dcb-3363626c6018"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 10:34:27.455628 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.455615    4473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/031e6c9b-93f8-458b-a20d-24927d6fb383-kube-api-access-b2jtg" pod "031e6c9b-93f8-458b-a20d-24927d6fb383" (UID: "031e6c9b-93f8-458b-a20d-24927d6fb383"). InnerVolumeSpecName "kube-api-access-b2jtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 10:34:27.552186 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.552106    4473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/031e6c9b-93f8-458b-a20d-24927d6fb383-workload-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 10:34:27.552186 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.552134    4473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-istio-token\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 10:34:27.552186 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.552146    4473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-istio-data\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 10:34:27.552186 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.552154    4473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-istiod-ca-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 10:34:27.552186 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.552164    4473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-istio-podinfo\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 10:34:27.552186 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.552174    4473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/031e6c9b-93f8-458b-a20d-24927d6fb383-istiod-ca-cert\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 10:34:27.552186 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.552183    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dfqv\" (UniqueName: \"kubernetes.io/projected/fb74ea5d-0246-4fe2-8dcb-3363626c6018-kube-api-access-9dfqv\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 10:34:27.552186 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.552193    4473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea5d-0246-4fe2-8dcb-3363626c6018-credential-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 10:34:27.552566 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.552202    4473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea5d-0246-4fe2-8dcb-3363626c6018-istio-envoy\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 10:34:27.552566 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.552210    4473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb74ea5d-0246-4fe2-8dcb-3363626c6018-istio-token\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 10:34:27.552566 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.552218    4473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/031e6c9b-93f8-458b-a20d-24927d6fb383-istio-podinfo\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 10:34:27.552566 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.552227    4473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea5d-0246-4fe2-8dcb-3363626c6018-istio-data\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 10:34:27.552566 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.552234    4473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/031e6c9b-93f8-458b-a20d-24927d6fb383-credential-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 10:34:27.552566 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.552244    4473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/031e6c9b-93f8-458b-a20d-24927d6fb383-istio-token\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 10:34:27.552566 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.552254    4473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/031e6c9b-93f8-458b-a20d-24927d6fb383-istio-envoy\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 10:34:27.552566 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.552262    4473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-workload-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 10:34:27.552566 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.552270    4473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-istio-envoy\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 10:34:27.552566 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.552279    4473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-credential-socket\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 10:34:27.552566 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.552287    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2jtg\" (UniqueName: \"kubernetes.io/projected/031e6c9b-93f8-458b-a20d-24927d6fb383-kube-api-access-b2jtg\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 10:34:27.552566 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.552295    4473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb74ea5d-0246-4fe2-8dcb-3363626c6018-workload-certs\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 10:34:27.552566 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.552305    4473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb74ea5d-0246-4fe2-8dcb-3363626c6018-istio-podinfo\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 10:34:27.552566 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.552313    4473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jn5n\" (UniqueName: \"kubernetes.io/projected/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c-kube-api-access-2jn5n\") on node \"ip-10-0-139-73.us-west-1.compute.internal\" DevicePath \"\""
Nov 03 10:34:27.981662 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.981627    4473 generic.go:356] "Generic (PLEG): container finished" podID="1b80f43c-c1d9-4f07-84eb-7aa145b3d03c" containerID="7e0e27efcf2d0388eb1dc4d9013c9b575739d7fd33a1f1e391294b6ab2775852" exitCode=0
Nov 03 10:34:27.981877 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.981677    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"
Nov 03 10:34:27.981877 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.981718    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl" event={"ID":"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c","Type":"ContainerDied","Data":"7e0e27efcf2d0388eb1dc4d9013c9b575739d7fd33a1f1e391294b6ab2775852"}
Nov 03 10:34:27.981877 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.981789    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl" event={"ID":"1b80f43c-c1d9-4f07-84eb-7aa145b3d03c","Type":"ContainerDied","Data":"b42a30d197fb2ac0f478ef20622a36fc0c7a2a694b7c8709f206889aff8c9e06"}
Nov 03 10:34:27.981877 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.981823    4473 scope.go:122] "RemoveContainer" containerID="7e0e27efcf2d0388eb1dc4d9013c9b575739d7fd33a1f1e391294b6ab2775852"
Nov 03 10:34:27.983269 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.983248    4473 generic.go:356] "Generic (PLEG): container finished" podID="fb74ea5d-0246-4fe2-8dcb-3363626c6018" containerID="68c7cc705b268144f5d8c1d337e4d099257f0bc95573461fbc9ef043101602ed" exitCode=0
Nov 03 10:34:27.983397 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.983280    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq" event={"ID":"fb74ea5d-0246-4fe2-8dcb-3363626c6018","Type":"ContainerDied","Data":"68c7cc705b268144f5d8c1d337e4d099257f0bc95573461fbc9ef043101602ed"}
Nov 03 10:34:27.983397 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.983309    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"
Nov 03 10:34:27.983397 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.983329    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq" event={"ID":"fb74ea5d-0246-4fe2-8dcb-3363626c6018","Type":"ContainerDied","Data":"7f3ec5e6ee5233280677bcdb452068171b8b659bda8d349ee5d4906fcd307703"}
Nov 03 10:34:27.984794 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.984773    4473 generic.go:356] "Generic (PLEG): container finished" podID="031e6c9b-93f8-458b-a20d-24927d6fb383" containerID="67004f48df7494bae6905dac1007d5233f73b00c7808dad8610a54665aed4321" exitCode=0
Nov 03 10:34:27.984893 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.984807    4473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"
Nov 03 10:34:27.984968 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.984807    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs" event={"ID":"031e6c9b-93f8-458b-a20d-24927d6fb383","Type":"ContainerDied","Data":"67004f48df7494bae6905dac1007d5233f73b00c7808dad8610a54665aed4321"}
Nov 03 10:34:27.984968 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.984959    4473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs" event={"ID":"031e6c9b-93f8-458b-a20d-24927d6fb383","Type":"ContainerDied","Data":"7838ab31903353a3f30a0930029ab6da6a2bb8a382c789ecb7d75e6ccb5b01a4"}
Nov 03 10:34:27.991569 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.991551    4473 scope.go:122] "RemoveContainer" containerID="7e0e27efcf2d0388eb1dc4d9013c9b575739d7fd33a1f1e391294b6ab2775852"
Nov 03 10:34:27.991845 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:34:27.991826    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e0e27efcf2d0388eb1dc4d9013c9b575739d7fd33a1f1e391294b6ab2775852\": container with ID starting with 7e0e27efcf2d0388eb1dc4d9013c9b575739d7fd33a1f1e391294b6ab2775852 not found: ID does not exist" containerID="7e0e27efcf2d0388eb1dc4d9013c9b575739d7fd33a1f1e391294b6ab2775852"
Nov 03 10:34:27.991930 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.991856    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e0e27efcf2d0388eb1dc4d9013c9b575739d7fd33a1f1e391294b6ab2775852"} err="failed to get container status \"7e0e27efcf2d0388eb1dc4d9013c9b575739d7fd33a1f1e391294b6ab2775852\": rpc error: code = NotFound desc = could not find container \"7e0e27efcf2d0388eb1dc4d9013c9b575739d7fd33a1f1e391294b6ab2775852\": container with ID starting with 7e0e27efcf2d0388eb1dc4d9013c9b575739d7fd33a1f1e391294b6ab2775852 not found: ID does not exist"
Nov 03 10:34:27.991930 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.991878    4473 scope.go:122] "RemoveContainer" containerID="68c7cc705b268144f5d8c1d337e4d099257f0bc95573461fbc9ef043101602ed"
Nov 03 10:34:27.999613 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.999593    4473 scope.go:122] "RemoveContainer" containerID="68c7cc705b268144f5d8c1d337e4d099257f0bc95573461fbc9ef043101602ed"
Nov 03 10:34:27.999883 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:34:27.999865    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68c7cc705b268144f5d8c1d337e4d099257f0bc95573461fbc9ef043101602ed\": container with ID starting with 68c7cc705b268144f5d8c1d337e4d099257f0bc95573461fbc9ef043101602ed not found: ID does not exist" containerID="68c7cc705b268144f5d8c1d337e4d099257f0bc95573461fbc9ef043101602ed"
Nov 03 10:34:27.999930 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.999891    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68c7cc705b268144f5d8c1d337e4d099257f0bc95573461fbc9ef043101602ed"} err="failed to get container status \"68c7cc705b268144f5d8c1d337e4d099257f0bc95573461fbc9ef043101602ed\": rpc error: code = NotFound desc = could not find container \"68c7cc705b268144f5d8c1d337e4d099257f0bc95573461fbc9ef043101602ed\": container with ID starting with 68c7cc705b268144f5d8c1d337e4d099257f0bc95573461fbc9ef043101602ed not found: ID does not exist"
Nov 03 10:34:27.999930 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:27.999905    4473 scope.go:122] "RemoveContainer" containerID="67004f48df7494bae6905dac1007d5233f73b00c7808dad8610a54665aed4321"
Nov 03 10:34:28.007447 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:28.007433    4473 scope.go:122] "RemoveContainer" containerID="67004f48df7494bae6905dac1007d5233f73b00c7808dad8610a54665aed4321"
Nov 03 10:34:28.007684 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:34:28.007668    4473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67004f48df7494bae6905dac1007d5233f73b00c7808dad8610a54665aed4321\": container with ID starting with 67004f48df7494bae6905dac1007d5233f73b00c7808dad8610a54665aed4321 not found: ID does not exist" containerID="67004f48df7494bae6905dac1007d5233f73b00c7808dad8610a54665aed4321"
Nov 03 10:34:28.007740 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:28.007692    4473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67004f48df7494bae6905dac1007d5233f73b00c7808dad8610a54665aed4321"} err="failed to get container status \"67004f48df7494bae6905dac1007d5233f73b00c7808dad8610a54665aed4321\": rpc error: code = NotFound desc = could not find container \"67004f48df7494bae6905dac1007d5233f73b00c7808dad8610a54665aed4321\": container with ID starting with 67004f48df7494bae6905dac1007d5233f73b00c7808dad8610a54665aed4321 not found: ID does not exist"
Nov 03 10:34:28.150550 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:28.150475    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"]
Nov 03 10:34:28.200720 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:28.200685    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gvlpl"]
Nov 03 10:34:28.269975 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:28.269929    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"]
Nov 03 10:34:28.275631 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:28.275608    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-c7mlq"]
Nov 03 10:34:28.390220 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:28.390197    4473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"]
Nov 03 10:34:28.396603 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:28.396579    4473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9wsvs"]
Nov 03 10:34:29.867851 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:29.867822    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="031e6c9b-93f8-458b-a20d-24927d6fb383" path="/var/lib/kubelet/pods/031e6c9b-93f8-458b-a20d-24927d6fb383/volumes"
Nov 03 10:34:29.868383 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:29.868368    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b80f43c-c1d9-4f07-84eb-7aa145b3d03c" path="/var/lib/kubelet/pods/1b80f43c-c1d9-4f07-84eb-7aa145b3d03c/volumes"
Nov 03 10:34:29.868760 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:29.868747    4473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb74ea5d-0246-4fe2-8dcb-3363626c6018" path="/var/lib/kubelet/pods/fb74ea5d-0246-4fe2-8dcb-3363626c6018/volumes"
Nov 03 10:34:37.123050 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:37.123017    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 10:34:37.129566 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:37.129543    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bt4zt_0a5d9c55-0799-4ae1-a902-dc06f813907c/ovn-acl-logging/0.log"
Nov 03 10:34:46.936875 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:34:46.936826    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:34:49.349642 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:34:49.349615    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-xbn6s_a025cb89-c80d-48f7-9f0a-93c8b6c24950/limitador/0.log"
Nov 03 10:35:16.937601 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:35:16.937563    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"
Nov 03 10:35:36.306880 ip-10-0-139-73 kubenswrapper[4473]: I1103 10:35:36.306848    4473 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-xbn6s_a025cb89-c80d-48f7-9f0a-93c8b6c24950/limitador/0.log"
Nov 03 10:35:46.937560 ip-10-0-139-73 kubenswrapper[4473]: E1103 10:35:46.937358    4473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tft2z" containerName="extractor"